Building AI that works in the real world

How to evaluate AI features for validity, bias, and data quality

Cross-functional teams — including product, engineering, and patient safety experts — evaluate healthcare AI features for their intended use and real-world conditions. Consider a car that drives itself around a test track. Impressive, but the real question is how it handles a construction detour, poor visibility, or any real-world situation its developers didn’t anticipate.

Healthcare AI deserves the same attention to the conditions in which a product will be used. Thorough evaluation calls for a broad scope far beyond a single accuracy result. For example: does the solution do the intended job? Does performance differ across relevant patient groups? Is the underlying data fit for that use?

AI-specific evaluation adds to foundational controls such as patient safety review, software development requirements, information security, and contractual protections.

This is why successful AI evaluation should reflect a feature’s task and clinical risk, use defined standards before release, and continue after deployment. Thorough evaluation provides a foundation for trust by giving healthcare leaders evidence on which to base confidence in a feature’s intended use.

Start with the clinical task and what could go wrong

Consider a hypothetical visit in which a clinician discusses a medication change with a patient. An AI feature turns the encounter information into a draft note for the clinician to review. The draft reads smoothly, but it omits the medication change.

That example shows why evaluation needs to begin with the feature’s intended use. Producing readable text is only part of the job. The draft also needs to accurately capture information relevant to its documentation purpose.

Before testing, cross-functional teams need to identify which errors could matter in that workflow. For this draft-note feature, they might ask:

• Does the draft preserve important information discussed during the encounter?
• Who reviews it, and when can they identify and correct an omission?
• If the omission remains in the approved note, could it affect how another clinician understands the encounter or uses the record later?

Human review is part of the risk assessment, but it doesn’t remove the need to evaluate the quality of the draft itself. Understanding the task, clinical context, oversight, and potential consequences helps teams establish what to measure across validity, bias, and data quality.

Evaluate the feature against three critical dimensions: Validity, bias, and data quality

Validity, bias, and data quality form a baseline for evaluating an AI feature. Together, they help teams examine performance more thoroughly than one overall accuracy result, while remaining part of a broader evaluation process. Depending on its intended use and risks, a feature may also need additional evaluation to assess reliability, safety, and security.

Validity: Does it do what it is supposed to do?

Validity means the feature performs its intended task correctly and reliably enough for its defined use.

For example, do the AI-generated draft visit notes accurately represent the encounter information needed for their intended documentation purpose? Are they concise enough to make relevant information easy to find, without omitting important details? A note can be factually accurate but still contain unnecessary detail or repetition that makes it harder for a clinician to use.

Examining errors that matter for the task helps teams assess whether the feature performs adequately for its intended use and make improvements when needed. These findings support confidence in its usefulness and reliability.

Bias: Does it perform consistently across patient demographics?

An AI feature should perform well across all patient groups. In other words, an overall result can hide weaker performance for a smaller patient group and may not provide sufficient grounds for confidence across all relevant groups. For example, finding that feature performs well for patients ages 35-44 doesn’t necessarily mean it does as well for patients ages 65 and older.

Product teams therefore need to look beyond combined results and examine performance across patient groups relevant to the feature’s intended use. Does performance differ across those groups? Are any differences clinically significant?

Measuring differences helps a team investigate potential bias, but it doesn’t prove a feature is free of bias. The feature needs evaluation, even when a clinician reviews its output. A clinician may be able to correct a draft, but that doesn’t confirm whether the draft is consistently good across patient groups. Also, observing a performance difference does not by itself establish its cause; it gives the team something to investigate.

Data quality: Is the data fit for the feature’s intended use?

Product teams need to assess both the information a feature relies on, and the information used to evaluate it. For a draft-note feature, source information might include a transcript of the encounter. Evaluation data might include a set of encounters and reference materials used to assess the resulting drafts.

Missing, inaccurate, or ambiguous source information can affect a feature’s output. Evaluation data that poorly represent the intended populations and conditions can limit what the results tell a team about performance.

This is why evaluation data must be complete, representative, and reliable:  

• Complete: Does the information include what is needed for the task? Missing relevant information can limit both performance and assessment.
• Representative: Does the evaluation include the populations, settings, and workflows where the feature is intended to operate?
• Reliable: Can the team depend on the information’s accuracy and consistency? For example, with a draft-note, this could mean checking that the reference material used in the AI-generated note accurately reflects the original encounter.

Teams also need to understand what the evaluation does not establish. Evidence from one specialty may provide limited insight into performance in another. Limited representation of a patient group may make conclusions about that group less certain. Confidence in the findings depends partly on whether the evidence is reliable and relevant to the intended use.

Healthcare environments change. Assessments made during development need to be continued for the life of the feature, as it expands to new patient populations or the information entering the workflow changes.

A threshold is what turns evaluation into governance

A threshold is the defined boundary for meeting an applicable performance requirement. Teams establish those requirements before formal evaluation, measure performance, and compare the findings with the thresholds. Using that evidence to decide whether a feature is ready to advance connects evaluation to governance.

For a hypothetical draft-note feature, a team might evaluate whether notes are concise while preserving important information. Findings about unnecessary detail, repetition, or omissions can help the team identify where the feature needs improvement. After making changes, the team can evaluate again to assess whether those changes improved performance. This gives evaluation an ongoing role in product development: identifying weaknesses, guiding improvements, and measuring progress.

The three baseline evaluations — validity, bias, and data quality — work together. Teams should consider those findings alongside any additional evaluations required for the feature’s intended use and risks. A product team should define how it will interpret the evidence, investigate material shortcomings, and use the findings when deciding whether a feature is ready to advance. Meeting one requirement does not establish overall readiness, and AI-specific performance requirements remain alongside broader patient safety and other controls.

As the preceding article on AI governance explored, governance becomes concrete through the decisions it produces. Defined requirements give evaluation findings a consequence for product development and release, contributing to a more trustworthy process.

Evaluation doesn't stop when the feature launches

Pre-release evaluation provides evidence about performance under the conditions examined at that time, but the most important thing is how a feature performs in the real world. After deployment, a feature can encounter conditions that were not fully represented in that evaluation. Changes can occur in patient populations, specialties, workflows, or source information. Updates to the feature itself — such as changes to a draft-note feature’s underlying model or instructions — can also affect its output, even when the clinical workflow stays the same. Teams should re-evaluate performance against relevant requirements before releasing those updates. Monitoring helps teams identify when further investigation or evaluation is needed.

This continues the governance process after release: teams keep checking performance and use what they learn to inform decisions as real-world use evolves.

What this looks like in practice  

In practice, evaluation steps are connected rather than independent checkboxes. The task and risk shape the standards; evaluation provides evidence about whether the feature meets the standards, which informs readiness; and monitoring checks whether performance continues to meet expectations and signals when it’s time for further evaluation.

Good AI evaluation is built to evolve

As AI features and healthcare workflows change, evaluation needs to keep pace. Findings from real-world use can reveal gaps, prompt further testing, and inform product improvements.

Evaluating validity, bias, and data quality gives teams evidence to guide development and readiness decisions. Continuing that evaluation after deployment helps them understand whether performance still meets expectations. Together, these practices contribute to a more trustworthy development process — and a stronger basis for confidence in the product itself.

For healthcare leaders, that means looking for evidence of how a vendor sets standards, evaluates performance, and acts on what it learns. Trust grows when those commitments shape decisions throughout a feature’s lifecycle.

Learn more about AI-native healthcare technology within athenaOne®.