The Algorithm on the Stand: Risk Assessment Tools, Racial Bias, and the Future of Sentencing

In 2013, a man named Eric Loomis stood before a judge in Wisconsin and was sentenced to six years in prison. The judge cited many factors. One of them was a score — a number generated by a computer program called COMPAS, which had assessed Loomis as high risk for reoffending.

Loomis had never seen the algorithm. He had no way to challenge it. He did not know what questions it asked, how it weighed the answers, or why it reached the conclusion it did. The company that built it considered the formula a trade secret.

He appealed all the way to the Wisconsin Supreme Court, arguing that using a secret algorithm in his sentencing violated his constitutional rights. He lost.

This is where law, science, and justice collide — and where Mind on Trial's three-lens framework has the most to say.

What Are Risk Assessment Tools?

Risk assessment tools are software programs used by judges, parole boards, and prosecutors to predict how likely someone is to commit another crime in the future. The most widely used one is called

COMPAS — Correctional Offender Management Profiling for Alternative Sanctions. It was built by a private company called Northpointe, now known as Equivant.

COMPAS works by asking defendants a series of questions — about their criminal history, family background, education, finances, and social environment. It combines the answers into a score between one and ten. A high score means high risk. That score can influence whether someone is detained before trial, what sentence they receive, and whether they are granted parole. [1]

On the surface, this sounds like progress. Remove human bias. Replace gut instinct with data. Make the system more consistent and objective.

The problem is that the data is not objective. And in 2016, a team of investigative journalists proved it.

Lens One – The Scientific Evidence

In 2016, ProPublica published an investigation called "Machine Bias." Journalists Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner analyzed COMPAS scores for more than 7,000 people arrested in Broward County, Florida. They then tracked those individuals for two years to see what actually happened. [2]

Their findings were striking. Black defendants were nearly twice as likely as white defendants to be falsely flagged as high risk — meaning COMPAS predicted they would reoffend, and they did not. White defendants were more likely to be incorrectly labeled low risk when they went on to commit new crimes. [2]

The company disputed the findings, arguing that the tool was equally accurate across racial groups when measured a different way. But independent researchers kept looking.

In 2018, two computer scientists — Julia Dressel and Hany Farid — published a study in the journal Science Advances. They found that COMPAS was no more accurate at predicting recidivism than a group of random people recruited online with no criminal justice training whatsoever. Both the algorithm and the untrained humans were correct about 65 percent of the time. [3]

Sixty-five percent. For a tool influencing prison sentences.

The scientific evidence raises a fundamental question: if the tool is barely better than chance, and it produces racially skewed errors, what exactly is it doing in a courtroom?

Lens Two – The Neuroscience

To understand why these tools produce biased results, you have to understand something about how bias works — both in humans and in machines.

COMPAS was trained on historical criminal justice data. That data reflects decades of policing decisions, prosecution choices, and sentencing patterns — all of which research has shown to be influenced by racial bias at multiple levels. [4] When you train an algorithm on biased data, the algorithm learns the bias. It does not see race directly, in most versions, but it sees variables that are closely correlated with race — neighborhood, employment history, income, family background — and it uses those proxies to reach conclusions that mirror the original bias. This is called algorithmic bias, and it is not a flaw in the code. It is a reflection of the world the code was trained on. [5]

But there is a second layer here that goes deeper than the algorithm itself. It concerns the human brain.

When a judge is handed a risk score, research in cognitive neuroscience shows something troubling happens. Numerical scores have an anchoring effect on human judgment — they create a reference point that is very difficult to ignore, even when people are told they should think independently. [6] A judge who sees a score of nine out of ten does not process that number as one input among many. The brain treats it as a fact. It shifts the burden of proof invisibly. Now the defendant must overcome the number, not the other way around.

This is especially dangerous when the number carries hidden racial patterns. The judge does not see the bias in the algorithm. The judge sees a score. The score feels neutral. The brain treats neutral-seeming information as more credible, not less. The bias becomes invisible at exactly the moment when it is doing the most damage.

Lens Three – The Legal Interpretation

The legal questions surrounding risk assessment tools are serious and still largely unresolved.

The first question is due process. In Loomis v. Wisconsin, the defendant argued that using a proprietary algorithm — one whose internal logic could not be examined or challenged — violated his right to due process under the Fourteenth Amendment. The Wisconsin Supreme Court disagreed, ruling that COMPAS was just one factor among many and that the sentence could be justified without it. [7]

But legal scholars have pushed back hard on that reasoning. If a score influences a sentence, and the defendant cannot see how that score was calculated, they cannot meaningfully challenge it. Transparency is not a technicality — it is the foundation of a fair proceeding.

The second question is equal protection. The Fourteenth Amendment also prohibits states from denying equal protection of the law. If a tool systematically produces harsher predictions for Black defendants, and those predictions influence sentences, that is not an abstract statistical problem. It is a constitutional one. No court has yet ruled squarely on this, but the argument is gaining traction in legal scholarship. [8]

The third question is what sentencing is supposed to do. The law has always held that punishment should be based on what a person did — not on a statistical prediction of what they might do in the future. Risk assessment tools shift sentencing toward prediction rather than accountability. That is a quiet but significant change in what the criminal justice system believes it is for.

Putting All Three Lenses Together

The scientific evidence says the tools are barely more accurate than chance and produce racially skewed errors. The neuroscience says the way human brains process numerical scores makes those errors invisible and sticky. The law has not yet decided what to do with either of those facts.

That combination is exactly what makes this problem so hard — and so urgent.

Some states have begun restricting the use of risk assessment tools in sentencing. Others have moved toward open-source algorithms whose formulas can be publicly examined. Advocates argue that no predictive tool should be used in sentencing at all until these questions are resolved. [9]

Mind on Trial's position is straightforward: a tool that cannot be examined, challenged, or fully understood has no place in a proceeding where a person's freedom is at stake. Transparency is not optional. And a system that encodes historical bias into future sentences is not delivering justice — it is automating injustice at scale.

The next post in this series will look at another place where the legal system's assumptions collide with what neuroscience actually knows: eyewitness memory, and why the most trusted testimony in a courtroom is also one of the most unreliable.

Citations

[1] Northpointe / Equivant — COMPAS overview: equivant.com

[2] ProPublica, "Machine Bias" — Angwin et al., 2016: propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing

[3] Dressel & Farid, Science Advances, 2018: advances.sciencemag.org/content/4/1/eaao5580

[4] The Sentencing Project — Racial Disparities in Criminal Justice: sentencingproject.org

[5] MIT Technology Review — "Bias detectives: the researchers trying to make algorithms fair": technologyreview.com

[6] Tversky & Kahneman, anchoring and adjustment heuristic — foundational cognitive science: available via psycnet.apa.org

[7] Loomis v. Wisconsin, 881 N.W.2d 749 (Wis. 2016): law.justia.com/cases/wisconsin/supreme-court/2016/2015ap157-cr.html

[8] Harvard Law Review, "Algorithmic Accountability" — law.harvard.edu

[9] Brennan Center for Justice, "Risk Assessment in Criminal Sentencing": brennancenter.org