Conversational AI in healthcare: where it works, and where it is still unsafe

Conversational AI in healthcare is safe today for administrative work a human reads before it counts: scheduling, intake, insurance checks and documentation a clinician signs. It is not safe for triage, symptom advice or medication guidance, because those outputs are clinical judgements and, under FDA and MHRA rules, usually make the software a medical device.

This article is not clinical or legal advice. Every obligation below is stated with its jurisdiction and its date, and you should check the current text with your regulatory lead before you build against it.

What conversational AI in healthcare actually covers

Conversational AI in healthcare means any system that takes natural language from a patient, a clinician or an administrator, interprets it, and produces language or structured data in reply. That covers a patient-facing chatbot on a hospital website, a voice agent that answers the appointments line, an ambient scribe sitting in a consultation room, and a retrieval system a clinician queries in plain English against a patient record.

Four terms get used loosely and need pinning down, because the regulatory answer depends on which one you mean.

An electronic health record, EHR, is the longitudinal clinical record for a patient. An electronic medical record, EMR, is the record held by one practice or hospital. In the UK the equivalent phrase is the electronic patient record. HL7 version 2 is the older message standard most hospital interfaces still speak; FHIR is the modern resource-based standard, and SMART on FHIR is the authorisation pattern that lets a third-party application launch inside the record with a scoped token rather than a shared service account. If your conversational system cannot write through one of those, it is not integrated. It is a transcript generator with a copy-paste step.

Ambient documentation, often called an ambient scribe or ambient voice technology, is a product that listens to a clinical encounter and produces a transcript, a summary, or a draft note. Clinical documentation is the wider category, including dictation, templated notes and discharge letters. Clinical decision support is a different thing again: software that gives a clinician a recommendation about prevention, diagnosis or treatment. The three are frequently sold under one product name, and that bundling is where most regulatory trouble starts.

Protected health information, PHI, is the US term for individually identifiable health information held by a covered entity or its business associate. De-identification under 45 CFR 164.514(b)(2) means removing eighteen categories of identifier, and voice prints are explicitly one of them, which matters if you plan to retain audio. A business associate agreement is the contract that extends HIPAA obligations to a vendor processing PHI on a covered entity's behalf. In the UK and EU the governing framework is UK GDPR or the GDPR, with health data as a special category and data residency a contractual question rather than a statutory one.

Software as a medical device, SaMD, is software intended for a medical purpose that is not part of a hardware device. Whether your product is SaMD is the question that decides whether you need FDA clearance in the US, a UKCA mark through an approved body in Great Britain, or CE marking under the EU medical devices regulation. Nothing about the model architecture decides it. Intended purpose decides it.

The line is drawn by the failure mode, not by the model

Healthcare AI content usually sorts use cases by how impressive they are. That is the wrong axis. Sort them by what happens when the system is confidently wrong and nobody catches it.

A scheduling agent that mishears a date books the wrong slot. A patient rings back. The cost is an hour of someone's time. A summarisation model that drops a stated allergy from a draft note produces a record error that a clinician may or may not spot before signing, and that error then propagates into every downstream decision that reads the record. A symptom checker that tells a patient with a subarachnoid haemorrhage to rest and take paracetamol produces a death. Same underlying technology in all three. The distance from the output to an irreversible action on a human body is what changed.

Regulators encode that distance rather than the technology. The MHRA said so directly in its guidance on ambient voice technology-enabled products, published 29 July 2026: "a product's intended purpose is not specifically determined by the inclusion of any particular technology". The same document notes that general disclaimers do not help, because "general disclaimers (for example 'this product is not for diagnosis') are not acceptable to demonstrate a product is not a medical device if medical claims are made or implied elsewhere in the product labelling or associated promotional material".

There is a second reason capability is the wrong axis. Current systems cannot reliably signal their own uncertainty on clinical content. A language model will produce a fluent, well-formatted, confidently worded differential diagnosis whether or not the evidence in the record supports it, and the confidence expressed in the text carries no information about whether the answer is right. The MHRA names the specific hazard in the same guidance: "hallucination is a well-known behaviour of GenAI with potentially broad impact in AVT products", and expects manufacturers to mitigate it by design rather than by warning labels.

That combination, an output that reads as authoritative and a system that cannot flag its own weak cases, is what makes clinical automation different from every other kind. Conversational AI in financial services has a bad day when a customer is quoted the wrong balance and calls back. The asymmetry is the whole argument, and it is why we treat the two domains differently in the same codebase.

So the line we use, and the line the regulators use, comes down to one question. Does a qualified human read the output and take responsibility for it before anything happens to the patient? If yes, you are in administrative or documentation territory and the risk is manageable with a review step and an audit log. If no, you are building a medical device, whether or not you called it one.

Layer one: administrative tasks a human reads

The administrative layer is where conversational AI for healthcare earns its keep today, and it is the layer most organisations skip past because it sounds unglamorous.

Appointment scheduling is the clearest case. A voice or chat agent that offers slots, books, reschedules and cancels handles no clinical content at all. It reads a diary, writes a diary, and confirms to a patient. The failure mode is a wrong booking, visible immediately, reversible by a phone call. Nothing about it engages medical device law in any jurisdiction, and the governing obligations are data protection ones: lawful basis, minimum necessary access to the demographic record, and a retention policy for the call recording.

Patient intake is the second. A conversational form that collects history, current medications, allergies and the reason for attendance before the appointment is handling clinical content, but it is not interpreting it. The system transcribes what the patient says into structured fields that the clinician reads at the encounter. The moment the same form starts telling the patient what their symptoms suggest, it has left this layer, and that transition is usually made by a product manager rather than by a regulatory assessment.

Insurance verification, eligibility checking and prior authorisation chasing sit here too. So does most of the revenue cycle: claim status enquiries, denial triage by reason code, payment posting queries. These are text-processing tasks against payer systems, with a billing specialist reading the output. In US practices this is frequently the single largest pool of recoverable staff hours, and it carries no clinical risk because no clinical judgement is being made or implied.

Wayfinding, opening hours, preparation instructions for a procedure read verbatim from an approved patient information leaflet, and repeat prescription request intake all belong in the same bucket, with one caveat on the last. Taking a request is administrative. Deciding whether to issue it is not.

What all of these share is not simplicity. Prior authorisation is not simple. They share the property that a competent human reads the output and owns the decision before it reaches the patient, and that the worst realistic failure costs time rather than health. That is the test worth applying before any healthcare automation project, and it is the reason our own conversational AI chatbot product is deployed in healthcare settings on the administrative side of the record rather than inside the consultation.

There is a practical argument for starting here that has nothing to do with safety. Administrative workflows give you real conversation logs, real integration experience against the record, and a governance track record with your information governance team, all before you have to defend anything clinical. Organisations that start in the clinical layer usually spend their first six months in an assurance process with nothing shipped.

Layer two: documentation, coding and summarisation

This layer touches clinical content, a qualified human decides, and the output must be traceable back to its source. It is the most commercially active part of healthcare conversational AI right now, and as of 29 July 2026 it also has the clearest regulatory guidance of any part of this field, in Great Britain at least.

The MHRA and NHS England published paired guidance on that date. The MHRA guidance on ambient voice technology-enabled products sets out which AVT functions qualify as medical devices under the UK Medical Devices Regulations 2002. The announcement accompanying it confirmed that products "intended solely for transcription, summarising of clinical conversations, drafting letters, or suggesting clinical codes for a clinician to review are not regulated as medical devices under the current framework", while products "intended to support diagnosis, treatment or prevention, or that take automated action such as placing orders without clinician review, are regulated as medical devices".

Read the five worked examples in that guidance rather than the summary, because the boundary sits inside them. A scribe producing a transcript is not a device. A scribe producing a summary for clinician review and correction before it is saved to the record is not a device. A product that formats transcript content into structured data for a clinician to accept or reject, including flagging potentially missing data, is not a device. A product suggesting clinical codes by matching terms explicitly said aloud during the encounter is not a device. A product drafting a discharge summary or letter from the record and the encounter, for clinician review, is not a device.

Then read the four examples that are devices. A summary product with a "generated insights" feature offering suggested diagnoses or treatment options is a device, even with a disclaimer in the interface. A functionally identical scribe whose manufacturer claims it "guides diagnosis and treatment planning and improves patient outcomes" is a device, because the claim creates the intended purpose. A product that analyses a transcript and record data to suggest diagnoses without presenting the reasoning is a device. An agent that saves notes to the record without clinician review and autonomously places follow-up test orders is a device.

Four of those five safe examples describe things we build. Our AI medical documentation bot does ambient capture, structured note drafting, ICD-10 and CPT code suggestion from what was said, and write-back to Epic, Cerner, Athenahealth and Allscripts through HL7 FHIR. It stays on the non-device side of that line for one reason: a clinician reviews, edits and submits every note, and the coding suggestions match terms present in the encounter rather than inferring conditions nobody mentioned. Remove the review step and the same product changes regulatory category.

What makes an ambient scribe a medical device?

Three things, in the MHRA's account, and only one of them is technical.

The first is the addition of derived clinical content. A summary restates what was said. An insight proposes something that was not said. The moment the output contains a diagnosis, a treatment option or a risk assessment that the clinician did not state, the product has a medical intended purpose regardless of how the feature is labelled in the interface.

The second is the marketing copy. This catches more products than the engineering does. The MHRA's Example 7 describes a scribe that is functionally identical to a compliant one, made a device purely by the manufacturer's claim that it guides diagnosis and treatment planning. Your website is part of your labelling. So is your sales deck.

The third is autonomy. A product that writes to the record without a human accepting the write, or that places orders on the strength of its own reading of the conversation, is taking actions a clinician would otherwise take. That is the MHRA's Example 8, and it is a device.

The practical consequence for anyone procuring in England is administrative rather than philosophical. NHS England's guidance on AI-enabled ambient scribing products, version 3 as at 29 July 2026, tells deploying organisations to assign a clinical safety officer, complete DCB0160 documentation with a safety case and hazard log, confirm the supplier has completed DCB0129, run a data protection impact assessment, and check the supplier against the NHS England AVT Supplier Registry, which requires a completed Digital Technology Assessment Criteria assessment. That is the actual cost of deploying a non-device product in the NHS, and it is not small. Vendors who tell you a non-device classification means no assurance work have not read the guidance.

Layer three: what should not be automated yet

Three categories, with the reason for each. The reason matters more than the verdict, because your product will sit near one of these boundaries and you will need to argue your own case.

Triage that gates access to care. A system that decides who is seen, in what order, or whether someone is seen at all, is making a decision with a clinical consequence and no human in front of it. Under the EU AI Act, emergency healthcare patient triage systems are named in Annex III point 5(d) as high-risk, alongside systems that evaluate and classify emergency calls or set dispatch priority. The European Commission's regulatory framework page confirms that, following the AI Omnibus Regulation which entered into force on 27 July 2026, obligations for Annex III high-risk systems apply from 2 December 2027, with product-embedded high-risk systems under Annex I, which includes medical devices, applying from 2 August 2028. The practical reading for a build starting today: if your roadmap includes triage in the EU, you are building a high-risk system and the compliance work is a scheduled cost, not a surprise.

Symptom advice without a guaranteed escalation path. The FDA position here is unambiguous and recent. Its Clinical Decision Support Software guidance, issued 29 January 2026 under docket FDA-2017-D-6569, states that "software functions that support or provide recommendations to patients or caregivers, not HCPs, meet the definition of a device". The non-device exclusion in section 520(o)(1)(E) of the FD&C Act is available only for software aimed at a licensed health care professional. A patient-facing symptom checker does not qualify for it. There is no version of that product in the US that is not a medical device, and a vendor selling one to a health system either holds a clearance or has not been assessed.

Medication guidance. Dose calculation, interaction checking presented as an answer, and titration advice are all treatment directives. The FDA's guidance is explicit that where a software function "provides a specific preventive, diagnostic or treatment output or directive", it fails Criterion 3 and is a device, and that FDA "does not consider software functions intended for a critical, time-sensitive task or decision to meet Criterion 4, because an HCP is unlikely to have sufficient time to independently review the basis of the recommendations".

That last clause is the one worth memorising, because it kills a design pattern people reach for constantly: the idea that adding a human review step makes anything safe. It does not, if the review is not realistically possible. An emergency physician with ninety seconds does not independently review the basis of a recommendation. The FDA names automation bias in the same section, describing it as "the propensity of humans to over-rely on a suggestion from an automated system", and notes that in situations requiring urgent action the bias increases.

Why is patient-facing triage treated differently from clinician-facing decision support?

Because the review step that makes clinician-facing support defensible does not exist on the patient side.

The FDA's four non-device criteria describe a chain of conditions. The software must not process images, in vitro diagnostic signals or patterns from a signal acquisition system. It must display, analyse or print medical information. It must support or provide recommendations to a health care professional without replacing or directing their judgement. And it must enable that professional to independently review the basis for the recommendation, which the guidance says requires the software or labelling to describe the algorithm's development and validation, the data it relied on, and the population it was validated against, in plain language.

A patient cannot perform that review. They have no training with which to assess whether a recommendation applies to them, no access to the validation data, and nobody to escalate to when the answer feels wrong. Remove the qualified reviewer and every safeguard in the non-device pathway collapses at once. This is why the exclusion is written for health care professionals and stops there.

There is a duty of care argument running alongside the regulatory one. A health system that deploys a symptom checker on its own domain is, in the eyes of a patient, giving advice. Whatever the terms of service say, that is a clinical relationship with a clinical duty attached, and no disclaimer relocates it to the vendor. The MHRA's position that disclaimers do not alter intended purpose points in exactly the same direction.

One honest qualification. Symptom-guided navigation is not the same as symptom advice. A system that asks what the problem is and routes the patient to the right service, without saying what the problem might be and with an unconditional escalation to a human for anything that could be urgent, is closer to scheduling than to diagnosis. That design is defensible. It requires a red-flag rule set written by clinicians, not a model deciding what counts as urgent, and it needs to fail towards a human every time it is unsure.

Working out where your own roadmap sits? Our generative AI engineering team runs a two-week boundary review that maps each planned feature to its regulatory classification and names the review step each one needs. You leave with a classification table you can hand to your information governance lead, not a proposal.

The safety boundary, task by task

This is the table we use internally before scoping anything in healthcare. Each row states how close the output sits to the patient, who reads it before it counts, what the regulatory position is, and what we would actually do.

Two things fall out of that table that are worth saying plainly. The first is that the verdict column is decided almost entirely by the third column. Where a qualified human reads the output and owns it, the task is deployable. Where nobody does, it is not, and no amount of model quality changes the answer.

The second is that the rows are not ordered by difficulty. Prior authorisation chasing is harder to engineer than a symptom checker. It is also safe, and the symptom checker is not.

What does an audit trail actually have to contain?

More than most teams expect, and the US has written down an unusually specific answer.

The certification criterion for decision support interventions at 45 CFR 170.315(b)(11) requires certified health IT to expose source attributes for what it calls a Predictive Decision Support Intervention, defined in 45 CFR 170.102 as technology supporting decision-making based on algorithms or models that derive relationships from training data and produce a prediction, classification, recommendation, evaluation or analysis. The source attribute requirements in the eCFR run to nine categories, and they are a good specification for any clinical AI whether or not certification applies to you.

They require the developer's name and contact details, the funding source for the implementation, a description of what the intervention outputs and whether that output is a prediction, classification, recommendation, evaluation or analysis. They require the intended use, the intended patient population, the intended user, and the intended decision-making role, with the regulation naming the options as informs, augments or replaces clinical management. They require cautioned out-of-scope uses, known risks, inappropriate settings and known limitations. They require the inclusion and exclusion criteria that shaped the training set, the demographic representativeness of that set, and the relevance of the training data to the setting where the system is actually deployed. They require an external validation description naming who conducted it and on what data. They require quantitative performance measures for validity and fairness, both on data from the training source and on external data. And they require the frequency at which validity and fairness are monitored after deployment, plus the update schedule.

Write that list against your own system and the gaps appear immediately. Most conversational healthcare products can answer the first three categories and go quiet at external validation.

Provenance is the operational half of this. Every clinical claim in a generated summary should be traceable to the span of the transcript or the record field it came from, and that link should survive into the stored artefact rather than living only in the generation request. In practice that means storing, per generated note: the model and version, the prompt template version, the retrieved record fields with their FHIR resource identifiers, the transcript offsets each assertion drew on, the clinician's edits as a diff, and the timestamp and identity of the signature. A clinician challenged about a note eighteen months later needs to be able to reconstruct what the system saw and what they changed.

The distinction between a log and an audit trail is that a log is written for you and an audit trail is written for someone who does not trust you. If you would not show it to a regulator investigating an incident, it is a log. Our position, stated in full below, is that no healthcare organisation should move into the clinical layer before it has the second kind.

Can you send protected health information to a general-purpose model?

Yes, under conditions, and the conditions are contractual before they are technical.

In the US, the model provider becomes a business associate when it processes PHI on a covered entity's behalf, which requires a business associate agreement in place before any PHI moves. Several major providers will sign one. Check what it actually covers: the specific endpoints, whether the zero-retention configuration is contractual or a console setting someone can toggle, whether abuse monitoring retains prompts, and whether the agreement survives the provider's own subprocessor changes.

De-identification is the alternative to a business associate agreement, not a supplement to it. Under 45 CFR 164.514(b)(2), safe harbour requires removing all eighteen identifier categories, including names, all geographic subdivisions smaller than a state, all date elements other than year, medical record numbers, device identifiers, biometric identifiers including voice prints, and full-face photographic images. Ambient capture makes this harder than it looks, because the audio itself is a biometric identifier and a clinical conversation names people, places and dates continuously. De-identifying a transcript after the fact is a best-effort exercise, and the expert determination route under 164.514(b)(1) requires a documented statistical analysis rather than a filter.

Data residency is a separate question from HIPAA, which does not require US storage. It comes from your own contracts, from NHS England's data security expectations for suppliers, and from UK GDPR and GDPR restrictions on international transfers. If your health system's information governance lead has committed to keeping data in-country, a US-hosted API breaks that commitment whatever the business associate agreement says. That is one of the two main reasons healthcare teams end up self-hosting. We have written separately on the economics of private LLM hosting, and the residency constraint usually decides it before the cost model does.

Consent is the piece most often handled badly. Recording a consultation is a processing activity the patient should know about, with a route to decline that does not disadvantage them. NHS England's ambient scribing guidance expects transparency about how information is used and shared in the setting. A poster in the waiting room is not consent. A verbal notice at the start of the encounter, logged against the record, is closer.

What breaks at the integration layer

Most healthcare conversational AI projects that fail do not fail on the model. They fail where the language meets the record.

Write-back is the first wall. Reading from an EHR through FHIR is usually straightforward. Writing a clinical note into it is governed by the vendor's app programme, the health system's interface team, and often an approval cycle measured in months. Epic, Cerner, Meditech and the UK primary care systems each have their own route, and in English general practice the IM1 pairing interface is the mechanism that actually matters. Scope this before you scope the model, because it sets your timeline.

Identity is the second. A conversational agent that books appointments needs to resolve a caller to a patient record without a clinician in the loop, and getting that wrong writes to the wrong chart. In England the Personal Demographics Service is the authoritative source and suppliers must be onboarded to it. In the US the matching problem is unsolved at national level and every system has its own heuristics. Build the failure path first: what happens when the match is ambiguous, and who the agent hands to.

Coding fidelity is the third. ICD-10 in the US, SNOMED CT in the NHS, CPT for procedures, and a mapping between them that is never quite clean. A code suggested from a term said aloud is auditable. A code inferred from context is a clinical judgement wearing an administrative costume, and it will eventually be examined by a payer or an auditor who asks how the system arrived at it.

Latency is the fourth and the most underrated. A scribe that takes four minutes to produce a draft is a scribe the clinician stops using, because the note gets written manually while they wait. Documentation tools live or die on whether the draft is ready before the clinician leaves the room. This is the same constraint that governs voice agents in any high-volume operation, and healthcare is stricter about it than most sectors because the next patient is already waiting.

What we would not build

We sell an AI clinical assistant, so the following is against our own commercial interest and we will say it anyway.

Most healthcare organisations should deploy conversational AI in the administrative layer first and stay out of the clinical layer until they have an audit trail they would show a regulator. Not a dashboard. A per-output record of what the system saw, what it produced, who reviewed it, what they changed, and which model version produced it, retained for the same period as the clinical record itself. If you cannot produce that for a note generated last March, you are not ready for anything closer to the patient.

We would not deploy the differential diagnosis or deterioration prediction features of our own clinical assistant as non-device software. They produce new clinical information, and deterioration prediction from vitals and monitoring telemetry takes signal input, which fails the FDA's first non-device criterion outright. Those features belong in a device pathway with a clearance behind them, or in a research setting with ethics approval. Any vendor shipping them as ordinary enterprise software, us included, is describing a product that has either been cleared or has not been assessed.

We would not accept a vendor's claim of HIPAA compliance as an answer. HIPAA compliance is not a certification. Ask for the business associate agreement, the subprocessor list, the retention configuration in writing, and the incident notification timeline. Ask what happens to the audio.

We would not build a patient-facing symptom checker for a health system in the US or the UK on current technology. The regulatory answer is settled and the clinical risk is asymmetric: the upside is deflected calls, the downside is a missed emergency. If the requirement is to reduce call volume, symptom-guided navigation with clinician-written red-flag rules and unconditional escalation does most of the work at a fraction of the risk.

We would not describe any of this as permanent. The MHRA published its ambient voice technology guidance on 29 July 2026, and the National Commission into the Regulation of AI in Healthcare published 44 recommendations for a future UK framework across a 119-page report on 10 September 2026, calling for proportionate lifecycle regulation, function-based assessment of multifunction products, and continuous post-deployment monitoring rather than one-off pre-market assessment. A cross-government response is still to come. The boundary in this article is where it sits in September 2026 and some of it will move, most likely towards function-level rather than whole-product classification, which would help rather than hinder anyone who has already separated their administrative functions from their clinical ones in the architecture.

If you are scoping a healthcare deployment now, the useful first artefact is not a business case. It is the classification table above with your own features written into it, one row each, with the review step named. Our agentic AI engineering practice builds that with clients before any code is written, or book a working session and bring the feature list you already have. For the architecture questions sitting underneath this, our enterprise conversational AI guide covers the platform decisions, and the vendor market splits between platforms, agencies and building it yourself.