Artificial IntelligenceArchive edition6 min read

Evaluating a document summarization system

Design summaries around the decisions a reader needs to make.

Luminous geometric nodes arranged in a layered lattice — Evaluating a document summarization system
Original ARTware computational illustration.

A long meeting record may include an approved decision, an idea that was considered and rejected, a task assigned to one person and an unresolved issue that prevents work from proceeding. A summary that mentions each topic but blurs those distinctions is not necessarily useful. It might be fluent, short and recognizably based on the meeting while giving a reader the wrong understanding of what to do next. Evaluating document summaries calls for a precise account of the information readers need. Length and readability can matter, but neither establishes that a summary preserves the meaning of its source.

Begin with a reader and a purpose

Meeting records may be consulted for several reasons. A project coordinator may need to find committed actions and their owners. A new team member may want to understand why a design was changed. A manager may only need to see the unresolved questions before the next discussion. A short summary cannot preserve every detail equally well. Choose the job the feature is expected to perform. An overview intended to support follow-up work should prioritize actions, decisions and dependencies. An account of the reasoning behind a decision may need more background, even if it includes fewer operational details.

State the expected categories of information in terms that can be checked against a document. In a project-summary feature, these could include approved decisions, next actions, explicit assignments, target dates and questions requiring follow-up. Define how missing information is represented. The absence of an owner in the notes is not permission to choose a plausible owner. Where the notes contain a proposed date that was never confirmed, a system should not present the proposal as an agreed deadline. These distinctions are important because the output is intended to guide subsequent work, not simply reproduce the general subject of the discussion.

Identify facts that should survive paraphrasing

Prepare a reference record for a representative document. Read it in full and mark the facts necessary for the defined reader task, together with their supporting passages. For example, notes might state that a team approved a revised navigation layout for the desktop version, but postponed a decision about mobile behavior. A useful summary retains both the approval and its scope. A summary claiming the new layout was approved for the entire website has changed the decision. Mark the condition as part of the required fact instead of recording merely that a layout was approved.

Pay attention to quantities, named responsibilities, dates and words that signal uncertainty or obligation. A document can distinguish a minimum from a target, a scheduled release from a suggested window or a required task from an optional experiment. The summary can paraphrase the wording while preserving those meanings. Reference facts do not have to form a second polished summary. A short set of independently checkable statements may provide a better evaluation basis, because reviewers can see which specific information was retained, which was changed and which disappeared.

Related reading: AI-assisted writing with accountable editing.

Separate omissions from factual changes

A generated summary might leave out a condition entirely, report it incorrectly or include the condition but associate it with a different action. These are not interchangeable errors. Suppose a meeting approves a prototype subject to collecting updated image files. A summary that merely says the prototype was approved omits the dependency. One stating that the images have already been collected invents an outcome. A third might say the dependency concerns a separate website, attaching authentic information to an unsupported subject. Reviewing each error according to its nature helps identify whether the output could mislead a reader who depends on it.

Completeness also needs a defined scope. Every summary omits something, and retaining every remark would defeat its purpose. When an editorial feature is intended to give a quick action overview, an interesting explanation of an earlier alternative may be legitimately excluded. Conversely, a single sentence assigning a task can be more important than several pages of discussion. Review omissions against the declared purpose rather than against an arbitrary proportion of the document. If reviewers disagree over which detail matters, clarify the expected reader task or add a note explaining the special importance of that detail.

Layered flowing curves with fine network-like intersections
ARTware visual study. A complementary computational composition.

Construct a collection that tests changes in status

Include documents in which positions evolve. A possible date may be mentioned at the beginning and confirmed later, or the participants may explicitly reject it. A decision may apply to one component but not another. Some records may contain a series of ideas but no approved outcome. Others may end abruptly before a discussion is resolved. A summarization system needs to represent the status of statements, not just recognize their vocabulary. Cases involving disagreement or incomplete records help reveal responses that turn every prominent proposal into a fact. Include ordinary well-organized notes as well, because those represent a major part of the intended workload.

References to earlier topics can be another source of difficulty. A sentence near the end stating that the second option was accepted may depend on a list of alternatives many paragraphs earlier. Mark both passages as relevant evidence. If a feature processes only part of a long document at a time, evaluate whether important relationships survive that arrangement. A local summary of the final section may mention the accepted option without identifying it. A local summary of the earlier section may explain the options without knowing the outcome. Combining such descriptions requires preserving the relationship between them, not merely concatenating fluent paragraphs.

Compare output configurations under consistent conditions

Use the same documents and reference facts to compare proposed settings. Record the accepted summary length or structure, the handling of long documents and any instructions about priorities. Have reviewers assess the individual expected facts rather than compare which version they find more elegant. Calculate separate observations for required information retained, required information distorted, unsupported assertions introduced and structural requirements met. One configuration may preserve more assignments but add more guessed dates. Another may produce nearly perfect short summaries of simple notes yet lose essential conditions in complex meetings. Looking at the failure patterns makes the difference actionable.

Consider how the interface will present the result. A compact list of actions can be convenient if each action is genuinely present in the notes. Attaching a source excerpt to an important item allows a reader to check its status or context. A narrative overview may better communicate the relationship between several decisions. Test the output in its intended presentation. If a display truncates a crucial qualification, the complete generated text does not solve the user's problem. If a reader cannot distinguish an open question from an accepted action, review the labels and structure as well as the wording.

Ask readers to perform the intended task

A factual evaluation establishes whether the summary is supported by its source. It does not by itself establish whether a person can use the summary efficiently. Give participants a representative task, such as identifying their next assignment and the dependency that must be resolved beforehand. Observe whether the summary lets them answer correctly without repeatedly searching the full notes. Ask which items remain unclear and which require a source check. Compare their experience with the existing method of locating that information. A short summary that forces users to read the entire document anyway may not be useful even if it contains no factual errors.

Maintain the evaluation as the summarization feature develops. Introduce new cases when the supported document types change or users begin relying on the output for a new purpose. Recheck previously important conditions after an update. A good result on a few carefully structured meeting notes should not be presented as proof of reliable handling of every kind of document. The useful conclusion is narrower: given a defined reader, a specific collection of material and clear information priorities, this configuration preserves the facts and distinctions necessary for the intended work with an acceptable level of unresolved uncertainty.

Explore the implementation area: AI solutions.