Tandem Health

Not always in a technical sense, but transparency about the product is increasingly expected. Many machine learning models cannot give a simple step-by-step reason for each output, so regulators focus on clear information for users: intended use, inputs, how the model was developed and tested, performance, and known limitations.

In the US, one of the criteria for clinical decision support to fall outside device regulation is that a healthcare professional can independently review the basis for its recommendations. FDA has also published transparency principles for machine learning devices with Health Canada and MHRA.

For buyers, a vendor's ability to document how its model works and where it fails is often more useful than heat maps or confidence scores alone.

An AI scribe typically works in four steps:

• Capture: a phone, computer or room microphone records the consultation after the patient is informed or consents.
• Transcription: speech recognition converts audio to text and separates speakers.
• Summarization: a language model extracts relevant content and drafts a note in the chosen format, such as a SOAP note, sometimes with codes or orders suggested.
• Review: the clinician edits and signs the draft, which is then filed in the EHR, often through an integration.

Each step can introduce errors, so final responsibility for the note stays with the clinician. Ask vendors whether audio is stored, whether data is used to train models, and how fabricated or missing content is measured.

Hallucination is when a generative AI tool produces content that sounds plausible but is false or unsupported by its inputs, such as a symptom, medication or exam finding that was never mentioned in a consultation.

It matters most in documentation, summarization and question-answering tools, because errors can be copied into the medical record. Common safeguards are:

• Clinician review and sign-off of every generated note or summary.
• Links from generated text back to the source audio or record.
• Vendor testing that measures omission and fabrication rates in realistic conditions.
• Clear policies on what the tool must not be used for.

WHO's guidance on large multi-modal models highlights inaccurate or false outputs as a key risk.