The Test That Flipped the Script on Police Bias

by Jon Scaccia April 9, 2025

What if everything you thought you knew about detecting discrimination was—statistically speaking—flawed?

That’s exactly what a group of researchers just uncovered when they applied a powerful new method to millions of police stops in California. And the results? Jaw-dropping. The test didn’t just suggest racial bias in policing—it confirmed it in a way we’ve never been able to before.

Even better? This test is so simple it could fit on a cocktail napkin. But behind its elegance lies a scientific shake-up that could change how we fight discrimination in everything from banking to hiring to education.

Let’s dive into the surprisingly wild world of the robust outcome test—aka, the statistical truth serum for systemic bias.

Why Traditional Tests Can Miss the Mark

Here’s the basic problem: For years, scientists and policymakers have relied on two main methods to detect bias:

1. Benchmark Test: Who gets searched, hired, admitted, etc.
2. Outcome Test: How successful those decisions are.

Sounds logical, right? If Black drivers are searched more often (benchmark), or if searches of Black drivers turn up less contraband than searches of white drivers (outcome), something’s fishy.

But there’s a sneaky issue baked into both tests: They can each get it wrong. Enter the statistical boogeyman: inframarginality. Basically, people making decisions (like cops or lenders) might be using the same rules across groups—but because of how risk plays out in real life, the numbers can look biased even when they’re not. Or worse, the numbers might not look biased, even when discrimination is happening.

It’s like trying to figure out who’s a better basketball player based only on who shoots more. Spoiler: that doesn’t tell the whole story.

The “Aha!” Moment: A Hybrid Solution

That’s where the researchers got clever.

They thought: What if we combine both the benchmark and outcome tests? Like peanut butter and chocolate, maybe the combo is better than either one alone.

And guess what? It worked. Really, really well.

The “robust outcome test” flags discrimination only when two things are true:

• A group is less likely to get the decision (e.g., fewer loans, more searches).
• But when they do get it, they have better outcomes (e.g., more likely to repay, less contraband found).

In other words, if Black drivers are searched more and those searches turn up less stuff, it’s a pretty solid sign they’re being held to a lower bar—aka, discrimination.

It’s like saying, “You barely let me play, but when I do, I score more points than anyone. What gives?”

Proof in the Police Stops: What California Data Revealed

The researchers put their method to the test with real data: 2.8 million police stops across California. (Yep, that’s million with an m.)

They crunched the numbers using their robust test and found something the old tests often missed:

👉 A consistent pattern of racial bias against Black and Hispanic drivers across dozens of law enforcement agencies.

In fact, the standard outcome test sometimes claimed that white drivers were being discriminated against—something that didn’t line up with any real-world evidence or common sense. That’s a red flag for a broken test.

The robust outcome test, however, cut through the noise and spotlighted the actual disparities. It was like switching from blurry goggles to 4K vision.

So What? Why This Matters (A Lot)

This new test isn’t just about policing. It could revolutionize how we detect bias in:

• 💰 Banking: Are minority borrowers being denied loans they could repay?
• 📚 Education: Are admissions offices holding certain students to unfair standards?
• ⚖️ Justice: Are judges or prosecutors playing favorites—consciously or not?

Because the robust outcome test works with just a few simple numbers (decision rates + outcome rates), it can be applied even when data is messy or incomplete—a huge bonus in the real world.

Plus, it doesn’t require knowing why a decision was made. It just looks at what happened, and whether the pattern is fair. And that’s a game-changer.

The Secret Sauce: A Gentle Assumption with Mighty Power

All this magic relies on a modest-sounding mathematical idea: the Monotone Likelihood Ratio Property (don’t worry, there won’t be a quiz). Basically, it assumes that the risk of success (like repaying a loan or having contraband) increases in a consistent way for everyone.

And here’s the cool part: The researchers checked this idea against real data in lending, education, and criminal justice—and it mostly held up. Even when it didn’t hold perfectly, the robust test still worked better than the old one.

So it’s not just a theoretical win. It works in practice.

Wrapping It Up: Science That Sparks Change

Here’s the takeaway: Sometimes, a small tweak in how we look at data can flip the story—and shift the spotlight onto long-overlooked truths.

With the robust outcome test, we now have a sharper, fairer way to identify discrimination. It doesn’t fix the problem on its own. But it does give communities, policymakers, and watchdogs a stronger foundation to build on.

Because when it comes to justice, evidence matters. And now, the evidence just got a lot more powerful.

Let’s Explore Together

This study doesn’t just live in spreadsheets and simulations—it lives in our schools, streets, and systems. And now, you know how it works.

🤔 How do you see this research affecting your community?
🧠 What would you test for bias if you had this tool?
🔥 What’s the most mind-blowing science fact you’ve learned lately?

Drop your thoughts in the comments or share this post with a friend who’s ready to nerd out on justice. The more we understand the data, the better we can shape a fairer world.

Fight for Science:

The stakes for science have never been higher. In today’s turbulent political climate, staying informed is critical. Subscribe to our weekly newsletter to get the latest discoveries, major breakthroughs, and stories that matter most. Designed for teachers and science enthusiasts, this free resource enhances your teaching and understanding of science in real time. Subscribe today to ensure science stays at the forefront of public conversation! If you liked this blog, please share it—your referrals help This Week in Science reach more people when it’s needed most.