Trust Calibration for AI Software Builders

Image by Annie Ruygt

Trust calibration is a concept from the world of human-machine interaction design, one that is super relevant to AI software builders. Trust calibration is the practice of aligning the level of trust that users have in our products with its actual capabilities.

If we build things that our users trust too blindly, we risk facilitating dangerous or destructive interactions that can permanently turn users off. If they don’t trust our product enough, it will feel useless or less capable than it actually is.

So what does trust calibration look like in practice and how do we achieve it? A 2023 study reviewed over 1000 papers on trust and trust calibration in human / automated systems (properly referenced at the end of this article). It holds some pretty eye-opening insights – and some inconvenient truths – for people building AI software. I’ve tried to extract just the juicy bits below.

Limiting Trust

Let’s begin with a critical point. There is a limit to how deeply we want users to trust our products. Designing for calibrated trust is the goal, not more trust at any cost. Shoddy trust calibration leads to two equally undesirable outcomes:

• Over-trust causes users to rely on AI systems in situations where they shouldn’t (I told my code assistant to fix a bug in prod and went to bed).
• Under-trust causes users to reject AI assistance even when it would be beneficial, resulting in reduced perception of value and increased user workload.

What does calibrated trust look like for your product? It’s important to understand that determining this is less about trying to diagram a set of abstract trust parameters and more about helping users develop accurate mental models of your product’s capabilities and limitations. In most cases, this requires thinking beyond the trust calibration mechanisms we default to, like confidence scores.

For example, Cursor’s most prominent trust calibration mechanism is its change suggestion highlighting. The code that the model suggests we change is highlighted in red, followed by suggested changes highlighted in green. This immediately communicates that “this is a suggestion, not a command.”

In contrast, Tesla’s Autopilot is a delegative system. It must calibrate trust differently through detailed capability explanations, clear operational boundaries (only on highways), and prominent disengagement alerts when conditions exceed system limits.

Building Cooperative Systems

Perhaps the most fundamental consideration in determining high level trust calibration objectives is deciding whether your project is designed to be a cooperative or a delegative tool.

Cooperative systems generally call for lower levels of trust because users can choose whether to accept or reject AI suggestions. But these systems also face a unique risk. It’s easy for over-trust to gradually transform user complacency into over-reliance, effectively transforming what we designed as a cooperative relationship into a delegative one, only without any of the required safeguards.

If you’re building a coding assistant, content generator, or design tool, implement visible “suggestion boundaries” which make it clear when the AI is offering ideas versus making decisions. Grammarly does this well by underlining suggestions rather than auto-correcting, and showing rationale on hover.

For higher-stakes interactions, consider introducing friction. Require explicit confirmation before applying AI suggestions to production code or publishing AI-generated content.

Building Delegative Systems

In contrast, users expect delegative systems to replace human action entirely. Blind trust in the system is a requirement for it to be considered valuable at all.

If you’re building automation tools, smart scheduling, or decision-making systems, invest heavily in capability communication and boundary setting. Calendly’s smart scheduling works because it clearly communicates what it will and won’t do (I’ll find times that work for both of us vs. I’ll reschedule your existing meetings). Build robust fallback mechanisms and make system limitations prominent in your onboarding.

Timing Is Everything

The study suggests that when we make trust calibrations is at least as important as how. There are three critical windows for trust calibration, each with their own opportunities and challenges.

• Pre-interaction calibration happens before users engage with the system. Docs and tutorials fall into this category. Setting expectations up front can prevent initial over-trust, which is disproportionally more difficult to correct later.

Pre-interaction calibrations could look like capability-focused onboarding that shows both successes and failures. Rather than just demonstrating perfect AI outputs, show users examples where the AI makes mistakes and how to catch them.

• During-interaction calibration is trust adjustment through real-time feedback. Dynamically updated cues improve trust calibration better than static displays, and adaptive calibration that responds to user behavior outperforms systems that display static information.

Build confidence indicators that are updated based on context, not just model confidence. For example, if you’re building a document AI, show higher confidence for standard document types the system has seen thousands of times, and lower confidence for unusual formats.

• Post-interaction calibration focuses on learning and adjustment that helps users understand successes and failures in the system after interactions. These aren’t reliable, since by the time users receive the information, their trust patterns are set and hard to change.

Post-interaction feedback can still be valuable for teaching. Create “reflection moments” after significant interactions. Midjourney does this by letting users rate image outputs, helping users learn what prompts work best while calibrating their expectations for future generations.

Trust is front-loaded and habit-driven. The most effective calibration happens before and during use, when expectations are still forming and behaviors can still be shifted. Any later and you’re mostly fighting entrenched patterns.

Performance vs. Process Information

Users can be guided through performance-oriented signals (what the system can do) or process-oriented signals (how it works). The real challenge is matching the right kind of explanation to the right user, at the right moment.

• Performance-oriented calibration focuses on communicating capability through mechanisms like reliability statistics, confidence scores, and clear capability boundaries.
• Process-oriented calibration offers detailed explanations of decision-making processes, breakdowns of which factors influenced decisions, and reasoning transparency.

Process transparency seems like the obvious go-to at first glance, but the effectiveness of process explanations varies wildly based on user expertise and domain knowledge. If we are designing for a set of users that may fall anywhere on this spectrum, we have to avoid creating information overload for novice users while providing sufficient information to expert users who want the detail.

The most effective systems in the study combined both approaches, providing layered information that allows users to access the level of detail most appropriate for their expertise and current needs.

Static vs. Adaptive Calibration

I really wanted to ignore this part, because it feels like the study’s authors are passive aggressively adding todos to my projects. In a nutshell, adaptive calibration – when a system actively monitors user behavior and adjusts its communication accordingly - is orders of magnitude more effective than static calibration while delivering the same information to every user, regardless of differences in expertise, trust propensity, or behavior.

Static calibration mechanisms are easy to build and maintain, which is why we like them. But the stark reality is that they put the burden of appropriate calibration entirely on our users. We’re making it their job to adapt their behaviour based on generic information.

This finding has zero respect for our time or mental health, but it also reveals a legit opportunity for clever builders to truly separate their product from the herd.

Practical adaptive calibration techniques

• Behavioral adaptation: Track how often users accept vs. reject suggestions and adjust confidence thresholds accordingly. If a user consistently rejects high-confidence suggestions, lower the threshold for showing uncertainty.
• Context awareness: Adjust trust signals based on use context. A writing AI might show higher confidence for grammar fixes than creative suggestions, or lower confidence late at night when users might be tired.
• Detect expertise: Users who frequently make sophisticated edits to AI output probably want more detailed explanations than those who typically accept entire file rewrites.

The Transparency Paradox

The idea that transparency and explainability can actually harm trust calibration is easily the point that hit me the hardest. While explanations can improve user understanding, they can also create information overload that reduces users’ ability to detect and correct trash output. What’s worse, explanations can create a whole new layer of trust calibration issues, with users over-trusting the explanation mechanism itself, rather than critically evaluating the actual output.

This suggests that quality over quantity should be our design philosophy when it comes to transparency. We should provide carefully crafted, relevant information rather than comprehensive but overwhelming detail. The goal should be enabling better decision-making rather than simply satisfying user curiosity about system internals.

Anthropomorphism and Unwarranted Trust

It seems obvious that we should make interactions with our AI project feel as human as possible. Well, it turns out that systems that appear more human-like through design, language, or interaction patterns are notoriously good at increasing user trust beyond actual system capabilities.

So it’s entirely possible that building more traditional human-computer interactions can actually make our AI projects safer to use and therefore, more user-friendly.

• Use tool-like language: Frame outputs as “analysis suggests” rather than “I think” or “I believe”
• Embrace machine-like precision: Show exact confidence percentages rather than human-like hedging (“I’m pretty sure that…)

Trust Falls Faster Than It Climbs

Nothing particularly groundbreaking here, but the findings are worth mentioning if only to reinforce what we think we know.

Early interactions are critically important. Users form mental models quickly and then react slowly to changes in system reliability.

More critically, trust drops much faster from system failures than it builds from successes. These asymmetries suggest that we should invest disproportionately in onboarding and first-use experiences, even if they come with higher development costs.

Measurement is an Opportunity for Innovation

The study revealed gaping voids where effective measurement mechanisms and protocols should be, for both researchers and builders. There is a clear need to move beyond simple user satisfaction metrics or adoption rates to developing measurement frameworks that can actively detect miscalibrated trust patterns.

The ideal measurement approach would combine multiple indicators. A few examples of viable indicators are:

• Behavioral signals: Track acceptance rates for different confidence levels. Well-calibrated trust should show higher acceptance rates for high-confidence outputs and lower rates for low-confidence ones.
• Context-specific metrics: Measure trust calibration separately for different use cases. Users might be well-calibrated for simple tasks but poorly calibrated for complex ones.
• User self-reporting: Regular pulse surveys asking "How confident are you in your ability to tell when this AI makes mistakes?” can reveal calibration gaps.

The Calibrated Conclusion

It’s clear, at least from this study, that there’s no universal formula, or single feature that will effectively calibrate trust. It’s up to every builder to define and understand their project’s trust goals and to balance timing, content, adaptivity, and transparency accordingly. That’s what makes it both hard and worth doing. Trust calibration has to be a core part of our product’s identity, not a piglet we only start chasing once it has escaped the barn.

The Study:

Magdalena Wischnewski, Nicole Krämer, and Emmanuel Müller. 2023. Measuring and Understanding Trust Calibrations for Automated Systems: A Survey of the State-Of-The-Art and Future Directions. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ‘23), April 23–28, 2023, Hamburg, Germany. ACM, New York, NY, USA 16 Pages. https://doi.org/10.1145/3544548.3581197