Symphony for Text Generation: Introducing a new evaluation benchmark for clinical note quality

Introducing a benchmark for clinical note quality

Every AI scribe promises a better clinical note, but until now there has been no common way to measure quality at scale.

Today, we are introducing a new benchmark for AI-generated clinical notes for developers building ambient documentation. It scores each note on what it keeps, what it leaves out, and what it adds, then compares notes side by side across eight clinical quality dimensions. It comes with MedConv, a public dataset of 300 clinical encounters in English, Danish, and German, so any team can run the same test.

To show the benchmark in practice, we've used the benchmark to compare Symphony for Text Generation with Heidi and Tandem Health, two ambient scribes built on general-purpose AI. All three systems wrote mostly accurate notes. The clearest difference was in what they left out. Symphony produced the most complete notes on every dataset, at about twice the speed.

$2.5 billion invested in AI scribes, but no standard way to measure quality

AI scribes are one of the fastest-growing categories in healthcare. In the last five years, roughly $2.5 billion of venture capital has gone into ambient documentation, and more than a dozen well-funded scribe companies now compete for the same clinicians. Adoption has moved just as fast, and studies are beginning to report gains in clinician experience and documentation time.

But what the industry still lacks is a standard way to measure whether the notes these tools write are good enough for the medical record. There is no shared dataset, no agreed set of quality metrics, and no controlled way to compare one scribe against another.

Billions have been invested in generating clinical notes, but almost nothing in measuring them.

So why are there so many scribes, but no way to evaluate them? This is partly because note quality is hard to measure. Clinicians and published studies both point to omissions as the most common error type in AI-generated notes, with the most potential to affect patients. Omissions are also the hardest errors to catch. An evaluator that only checks what a note contains will not see what it left out.

Comparisons between products have their own problems. Studies use different datasets, different input formats, and different configurations, so it is hard to tell whether a difference comes from the underlying AI or from something else. Surface metrics like BLEU and ROUGE do not help much either. A note with a reversed negation can score higher than a correct paraphrase.

The result is that teams building scribes, and health systems choosing one, often fall back on pilot surveys and anecdotes. Those can mix up the quality of the AI with the user interface, the workflow integration, or the preferences of one clinician on the pilot team.

We built the benchmark to answer one question: given the same encounter, how good is the note?

How the framework works

The framework scores every note in two complementary ways.

The first is three content measures, each scored by an LLM judge that checks the note statement by statement:

• Groundedness: is everything in the note supported by the consultation transcript?
• Completeness: does the note capture the clinically relevant content of a clinician-validated reference note?
• Conciseness: does the note stay focused on what the reference includes, without padding?

Completeness is the measure most evaluations miss. It checks the reference against the generated note, so it can see what a note left out.

The second is a side-by-side comparison. A judge reads two notes from the same encounter and picks the better one on clinical qualities such as accuracy, thoroughness, and usefulness, based on PDSQI-9, an established instrument for rating clinical notes. Clinicians reviewed a sample of these verdicts and agreed with the judge 86% of the time.

MedConv: a shared dataset for fair comparison

A fair comparison also needs every system to work from the same inputs. Public datasets for this are scarce, mostly in English, and narrow in clinical scope.

So we built MedConv.

MedConv is 300 synthetic clinical encounters, 100 each in English, Danish, and German, spanning more than 15 specialties and settings from general practice to inpatient and pre-hospital care. Each encounter pairs a realistic, multi-speaker consultation transcript with a reference note written and validated by clinicians. It is free to use: run your own note generation pipeline on it, compare against our results, and see exactly what your notes leave out. MedConv will be available on Hugging Face.

Together, the framework and MedConv make it possible to benchmark ambient documentation systematically, putting numbers on note quality that until now could only be judged through anecdotes and qualitative feedback.

The benchmark in practice: Symphony for Text Generation Engine vs. Heidi and Tandem Health

Symphony for Text Generation is the documentation engine behind the hundreds of ambient scribes built on Corti. Developers use it to generate notes, manage templates, and extract structured clinical facts from each encounter. We tested it against Heidi and Tandem Health, two ambient scribes built on general-purpose AI, on the same 412 encounters. Every system received identical transcripts and the same note structure, with its vendor's default settings.

The result: Symphony wrote more complete notes, with fewer unsupported details, in half the time.

Notes that capture more of the visit

Every system wrote notes that were well supported by the conversation. The difference was in what got left out. Symphony produced the most complete notes on every dataset, with the largest lead on the longer, multilingual MedConv encounters. For your users, that means fewer details clinicians have to add back by hand.

Grounded and preferred on clinical quality

Symphony's notes had about half as many statements flagged as unsupported or overly specific as either competitor. In side-by-side comparisons, its notes were preferred on accuracy, thoroughness, and usefulness.

Twice as fast

Symphony returned a note in under 10 seconds on average, about twice as fast as either scribe. In a real-time workflow, the note is ready seconds after the visit ends. See the Realtime FactsR documentation for how to set this up.

Tune the note to the clinical setting

Out of the box, Symphony is tuned for completeness. That is the right default for complex encounters, but a busy general practice clinic may want shorter notes. In the default comparison, Heidi and Tandem Health were judged more succinct.

Symphony templates let you change that without rebuilding anything. Each section has separate controls for what to include, writing style, and output format, and sections can be reused across templates.

We tested this with two small changes for a general practice template: a short instruction to keep only decision-relevant information, and a terser writing style. Symphony went from the least preferred system on succinctness to the most preferred against both competitors, and stayed ahead on thoroughness.

You control every instruction, change only what your setting needs, and measure the result. Both templates from the study are published, so you can start from them in the Corti Console or through the API.

Built by a lab that measures what it ships

Corti has published clinical AI research for close to a decade, and building our own evaluation tools is part of how we work.

• When we built Symphony for Speech-to-Text, we found that word error rate hid the errors that matter most in clinical audio, such as drug names, dosages, and units. So we developed Error Align, an open-source alignment algorithm that makes speech recognition errors easier to locate and attribute.
• We also released MedDictate and MedTerm, public datasets for testing medical dictation and terminology. Other teams now use them: ElevenLabs evaluated its medical speech model on both.

This benchmark brings the same analytical rigor and open approach to clinical note generation.

Get started

Symphony for Text Generation is available now through the Corti API and Console.

• Read the Text Generation documentation
• Get an API key and test the templates on your own transcripts
• Talk to our team about running the benchmark against your current product

If you want the full methodology, or want to benchmark your own note generation pipeline on MedConv using our evaluation framework, read the full research paper here.