Artificial IntelligenceArchive edition6 min read

Designing a retrieval-based assistant

Ground responses in available material and make missing evidence visible.

Luminous geometric nodes arranged in a layered lattice — Designing a retrieval-based assistant
Original ARTware computational illustration.

An organization's written handbook is likely to contain the information needed to answer many everyday questions. The difficulty may be finding the right passage rather than producing a new explanation. Someone asking how to replace a damaged piece of equipment will not necessarily know the title of the relevant policy document. A retrieval-based assistant can look for relevant information and present an answer in ordinary language. Its value depends on the relationship between the returned answer and the maintained material. If it cannot locate the applicable policy, a persuasive response is worse than an honest report that the necessary information was unavailable.

Start with the documents and their ownership

The source collection should be defined before building the query interface. In an internal help tool, sources might include a current equipment policy, a procedure for requesting replacements and a page describing exceptions. An old policy retained for historical reasons should not silently compete with the current procedure. Record useful properties such as a document's title, subject, version, status and review date. The assistant has a much easier job if those distinctions are available when it looks for material. A long directory of undifferentiated files is not the same thing as a maintained body of knowledge.

Different documents may be authoritative on different points. A general handbook may state that equipment remains company property, while the replacement procedure defines whether damaged equipment should be brought to a specific office. A directory may explain which office operates in a particular region. Identify which source is expected to answer which kind of question and consider how the collection will be updated. When owners revise material, old indexing records need to be replaced or marked accordingly. Otherwise the assistant can appear to have access to the current handbook while relying on a passage removed from it months earlier.

Turn documents into passages worth retrieving

Searchable units need to be small enough to distinguish neighboring subjects and large enough to make sense when retrieved individually. Splitting a procedure at arbitrary character intervals can separate a rule from its qualification. For example, a sentence stating that equipment can be returned by post means something different if the following paragraph restricts that option to remote workers. Consider the structure of the source. Section boundaries, descriptive headings and complete subsections provide meaningful places to divide content. Keep the document identity and heading hierarchy together with each passage so that its meaning can be reconstructed.

A single approach to passage length will not work equally well for every document. A table of approved options might require the header and several rows to be kept together. A narrative guide may be divided at subheadings. A glossary definition might be meaningful on its own but need to be supplied alongside a longer procedure that uses the defined term. Experiment with search results rather than choosing a length purely by convention. If a question retrieves a passage that contains a pronoun with no identifiable subject, the selected boundary probably needs attention. If every candidate contains an entire lengthy document, the retrieval stage has done little to narrow the evidence.

Related reading: Designing human review into automated workflows.

Understand why a relevant result was selected

Retrieval can use matching words, semantic representations of text, document properties or combinations of these methods. Word matching is useful for specific terms, document identifiers and distinctive phrases. Meaning-based matching can find a passage despite a difference in wording between the question and the source. Metadata can constrain the search to current material or to a subject that has been established by the interface. Whatever technique is chosen, study the actual returned candidates. An unrelated passage about closing an account may contain the word return several times, yet it does not answer a question about returning a physical device.

There is a difference between the position of a promising passage in a search result and the completeness of the information it provides. Suppose a question asks whether borrowed equipment can be collected after a person moves to a different office. One candidate describes equipment collection, while another lists rules for office transfers. Both may be needed. The retrieval step should allow the answer to draw on the pieces that are necessary, rather than expecting the first matching paragraph to settle every aspect. Equally, an answer may be incomplete if the collection contains only one of those documents. Allow the system to indicate that limitation.

Require the answer to distinguish evidence from inference

Layered flowing curves with fine network-like intersections
ARTware visual study. A complementary computational composition.

A response can paraphrase a clear source without losing its relationship to the document. If a procedure explicitly says returns must be arranged before the final day of employment, an answer can express that requirement in straightforward language. It should retain the timing and any exception that changes it. An answer saying equipment must be returned before the end of the month would introduce a different condition. Give the responding component a clear instruction to work with retrieved material and distinguish supported claims from matters the available material does not establish.

Consider presenting references to the corresponding internal sections alongside the response. A reference can be a document title and heading, with a means of opening the source if the person is entitled to do so. That provides a practical way to check details without requiring the reader to begin the search again. Do not confuse an attached source reference with proof of correctness: it matters whether the cited passage supports the particular assertion. Test whether the reader can use the cited information to verify each requirement. A reference to a general handbook is insufficient where the answer depends on a specific exception that appears nowhere in the retrieved text.

Make missing and conflicting information visible

A useful answer sometimes starts by saying that a question cannot be resolved from the current documents. If a worker asks whether an exception applies to a device not mentioned in the handbook, it is reasonable to state what the handbook covers and identify the missing point. Inventing a rule would create the false impression of an authoritative procedure. Distinguish the case in which no relevant passage was retrieved from the case in which relevant documents disagree. If the text contains competing versions and the system cannot determine which governs the situation, that is an important fact to surface to the user.

Follow-up questions may change the search. A reader could ask about returning equipment by courier and later clarify that the equipment has already been damaged. The clarification introduces a different condition, and the assistant should search for evidence about it rather than merely elaborating its first answer. Retrieving only at the beginning of the conversation can make later responses depend on an increasingly irrelevant selection of passages. Define how a follow-up question is interpreted with necessary context, while avoiding excessive material that is unrelated to the current question.

Measure the retrieval and the answer separately

Prepare a set of questions with known supporting passages and ask whether those passages appear in the returned candidate collection. This checks retrieval independently of the ability to explain the material. For answers, review the claims and compare them against the sources. Include questions that mention the policy in its exact wording, questions expressed in informal language, questions requiring more than one passage and questions whose answers are deliberately absent. Add examples involving superseded policies. An assistant should not be rewarded for confidently repeating an old instruction when the current document gives a different requirement.

The results can point to different changes. Relevant passages that are never retrieved suggest a problem with indexing, document boundaries or search selection. Correct material that appears in the candidate collection but is omitted from the response suggests an issue with how evidence is presented to the responding component or how the expected answer is defined. A correct reply accompanied by an irrelevant reference is a separate quality problem. Examine these failure types individually instead of reporting only the fraction of answers judged useful.

Keep this evaluation available as documents change. Replace expected answers deliberately when the corresponding policy changes, and check that formerly valid instructions no longer appear as current requirements. Give the source owners a practical way to report an uncovered problem. The assistant is part of a publishing and maintenance process: its reliability depends not just on text generation, but on the continuing ability to locate and interpret the material from which the organization expects answers to be drawn.

Explore the implementation area: AI solutions.