Build a Local Document Search Baseline Before an AI Assistant

Before building an AI assistant that answers questions from company documents, check whether you can reliably find the right source at all. A small local search experiment makes that problem visible. You choose a few authorized fictional documents, write questions with known source locations and inspect what the search returns. The exercise can reveal missing terminology, duplicate versions and questions that the collection cannot answer before a generated paragraph hides those weaknesses behind confident prose.

This guide uses a downloadable Python example that searches a tiny fictional document collection with SQLite full-text search. It makes no model calls, requires no API key and does not contact a network. It is a retrieval baseline, not a semantic search engine or a complete retrieval-augmented generation system. Its purpose is to give you concrete sources and observable failures to compare with a later AI proposal. The documents describe an invented service operation; they are not Dr. Connor Robertson client policies or evidence of measured consulting outcomes.

Define what the first experiment should answer

Choose a narrow operational question such as finding the documented owner of an appointment-change decision. The first experiment should return candidate source records, not authorize a change. Write down what a user must still do after receiving the result: open the record, confirm that it is current and decide whether it actually applies to the question. That human step matters because a keyword match can contain the right words while describing a different situation.

Keep the initial collection small enough to inspect completely. A few fictional records are enough to test the mechanics and demonstrate failure modes. Do not start by copying a shared drive containing years of mixed permissions into a new index. The search problem includes deciding what belongs in the collection. A technically successful index can still be unusable if it mixes drafts, obsolete procedures and records that the intended reader should not access.

For this example, the collection contains three short records: an appointment-change procedure, a visitor check-in procedure and an equipment-loan procedure. Each has a stable document identifier, title and body. The appointment procedure assigns a coordinator to prepare a proposal and a manager to approve changes affecting a promised arrival window. Those facts are fictional. They let you write a question with an answer location you can verify before you run a search.

Prepare records with explicit identities and versions

Use a CSV file with doc_id, title and body columns for the supplied example. Give each record a unique identifier that remains stable when you correct a typo. Avoid identifiers based only on position in a spreadsheet because sorting the sheet can change the apparent identity of a document. In a larger real experiment, you would also need version, status, effective date and owner fields. The tiny utility deliberately demonstrates retrieval rather than claiming to implement that entire document-management process.

Write the fictional body text in complete sentences. A record that says only “manager, appointment, approval” will match keywords but give the reader little to inspect. The example procedure should explain the condition under which approval is required and what remains the coordinator's responsibility. That distinction allows you to test whether a later summary preserves a qualification instead of turning every appointment change into a manager decision.

Do not assume a search title is enough evidence. Two procedures can share a title while applying to different locations or periods. Retain a source identity in every result and verify it against the collection. If you later replace the fictional records with authorized materials, keep the source-to-record mapping and the decision about which version is current. Retrieval quality cannot compensate for an unresolved question about the authority of the underlying document.

Run the supplied example and inspect its limits

Download the utility and fictional CSV into a local working folder. The example command is python3 local_search.py fictional-documents.csv appointment. The utility opens an in-memory database, loads the CSV, searches the supplied phrase and prints JSON results containing the document identifier, title and complete body. Because the database is in memory, it is rebuilt for each invocation and is not saved as a separate index file. The input CSV remains on disk until you remove it.

Your Python installation must include SQLite with the FTS5 extension. If that capability is absent, the utility should report a clear error rather than silently switching algorithms. The demonstration uses Python's sqlite3 interface and parameterized values. It treats the supplied query as one literal phrase for full-text matching rather than exposing the full query-language syntax. A quote in the input is escaped for that phrase representation before the parameter is submitted.

The example ranks matching records using SQLite's built-in ranking function and returns up to five. The accompanying test checks known result identities and empty-result behavior, not exact floating-point ranking scores. Scores can depend on the collection and implementation details. Treat the order as a retrieval aid. It is not a probability that a document is correct, current or applicable, and it does not provide an answer to the user's operational question.

Write questions with an answer location before testing

Create an evaluation sheet before you start experimenting with query wording. For each question, record its identifier, the user-style question, the expected source document and a short explanation of why that record is relevant. A question such as “Who approves a change to a promised arrival window?” should point to the fictional appointment procedure. You know the answer location because you read the record, not because a model suggested a plausible citation.

Include questions the collection cannot answer. For example, none of the three fictional records states a customer refund policy. Mark that question as unsupported by the collection. An empty search result is appropriate evidence of a gap in this tiny experiment, although an empty result in a larger system would not prove that no relevant source exists. The test should reward recognizing the limit instead of inventing an answer that sounds like a common business policy.

Keep the original user-style question even if you also try shorter search phrases. The tiny baseline may not match a full natural-language question because the words do not appear together in a source. That is an informative limitation. If you replace every question with a perfect document keyword and discard the original, you will make the search look easier than the intended use case. Record both the original question and each deliberately chosen query variant.

Separate exact wording from the underlying information need

Suppose the source says “appointment change” while a reader asks about “moving a visit.” A literal phrase search for moving a visit may return nothing. The failure does not establish that the document is missing. It shows a vocabulary mismatch between the user and the indexed text. A human can try appointment, read the result and judge whether it answers the original question. Record that intervention rather than counting the first search as an automatic success.

You can address some vocabulary gaps through clearer document titles, approved terminology notes or a search interface that supports explicit alternatives. A later semantic search approach might help with paraphrases, but that is a hypothesis to test on the same questions. Do not assume that adding embeddings or a model will solve every mismatch. It may retrieve a related but wrong record, especially when the collection contains several procedures with similar language.

Keep a list of observed wording differences from actual authorized users when you reach that stage. For the fictional exercise, invent a few realistic variants without generating dozens of cosmetic copies. Appointment change, moving a visit and adjusting a promised window are useful distinctions because they test different wording. Replacing one fictional company name with twenty others would add volume without examining a new retrieval problem. The evaluation set should earn its size through meaningful cases.

Inspect the complete retrieved record before summarizing

A result list is only the start of a source check. Read the full fictional appointment record and identify the condition attached to manager approval. The coordinator prepares a proposal; the manager approves changes affecting an existing promise. A generated answer that says “the manager handles all scheduling” would erase the coordinator's role and expand the approval condition. The source might have been retrieved correctly while the answer still became inaccurate.

For this baseline, write your own short answer with the document identifier attached. Then list any question the record leaves unresolved. Perhaps it does not state what happens when the manager is absent. That missing procedure is a genuine document question. The appropriate next step is to ask the document owner, not to let a model manufacture a backup approver. Retrieval and answer writing are separate stages, and each can fail independently.

If you later add a model, retain the retrieved text and the proposed answer together during evaluation. Check each material claim against the source. A link to a document is not sufficient if the document does not support the sentence. NIST's generative AI profile identifies the risk of confidently generated erroneous content, including apparent supporting material. In this exercise, source checking is an explicit reviewer task rather than an assumed benefit of attaching a citation.

Use small retrieval measures with honest denominators

For a question with one expected source, record whether that source appears among the first results you actually inspect. If your review window is the first three records, define success at three before reviewing the run. Do not change the window afterward to include a relevant document that happened to rank lower. For the three-record fictional collection, the measure is especially limited: almost any broad query could return much of the collection.

Suppose a hypothetical evaluation has eight supported questions and two unsupported questions. Six supported questions retrieve the expected source in the chosen window, while two do not. The supported-question retrieval result is six of eight. The unsupported questions need a separate review: did the system appropriately show no answerable source, or did it present a vaguely related record as if it settled the question? Do not blend these into a single number without explaining the scoring rule.

Also record the amount of irrelevant material a reviewer must read. A search that always returns every document may include the expected source, but it does not necessarily reduce work. Measure retrieval coverage and review burden separately. For an early experiment, a small table of question, returned identifiers, expected identifier and failure note is often sufficient. The denominator should describe your actual test collection rather than imply general performance across all company knowledge.

Introduce a conflicting version deliberately

After the basic example works, make a separate fictional copy of the collection and add an obsolete appointment procedure with a different approval owner. Mark it clearly in its title or body as obsolete for this exercise. Run appointment again and inspect whether both records appear. The search tool can retrieve matching text without knowing which document your organization considers authoritative. That is the lesson the conflict case is meant to expose.

Do not fix the demonstration by silently deleting the uncomfortable result from the evaluation log. Decide what a real collection policy would require. Perhaps only approved current documents should enter the reader-facing index, while obsolete versions remain in a separate archive. Perhaps a version resolver must identify the current record before results reach an answer generator. Those are design decisions to document and test. The tiny utility does not implement them for you.

Once you define an authority rule, repeat the same question and show the difference in the collection or filtering process. Keep the old and new test inputs separate so the change is visible. A better result may come from improved document governance rather than a better ranking algorithm. That distinction matters when deciding what to build next and who should own the work. Sometimes the useful project is cleaning a procedure library before adding AI.

Evaluate access boundaries before using real company documents

The fictional collection has no permission differences. A real collection often does. A staff member may be allowed to read a scheduling procedure without being allowed to read customer records or personnel notes. An index should not turn a shared technical search function into wider access than the source materials permit. This guide does not implement identity, authorization or a production document connector, and the demonstration should not be described as ready for those needs.

For an authorized real pilot, identify the intended reader group and which sources that group can access. Keep a record of what is indexed, where copies exist and how removed documents will be removed from the searchable collection. A read-only source connection can still create additional copies of sensitive text in indexes, logs or caches. Understand the proposed system's actual data path before treating the word read-only as a complete handling policy.

Start with materials that are appropriate for every pilot participant. Avoid introducing confidential mixed-permission folders simply to make the collection larger. If the proposed assistant must answer across different access levels, that becomes a separate design and verification requirement. Test what a user without permission can retrieve, not only what an authorized user can find. A successful answer for one user does not establish that the same system respects every access boundary.

Compare a later AI system with the same baseline questions

A useful comparison keeps the source collection, question set and expected source locations stable while you change the retrieval approach. If a later system uses semantic search, record the retrieved identifiers before evaluating generated answers. That lets you distinguish a retrieval improvement from a fluent rewrite of incomplete evidence. If both retrieval and generation change at once, retain enough intermediate output to understand which stage caused an observed difference.

Use at least one paraphrase question, one exact-term question, one unsupported question and one conflicting-version case. These examine different limitations. A system that handles paraphrases well may still choose an obsolete source. A system that cites the current source may still invent an absence procedure. No single demonstration establishes overall reliability. The point of the small baseline is to make those questions concrete enough to test, discuss and revise.

Compare the full review process as well as the output. How long does it take a person to locate the source, confirm authority and assess the answer? Does the new system make missing evidence easier to notice or harder? A fast generated paragraph can create more verification work if it contains several unsupported claims. Keep measured times separate from hypothetical planning estimates and avoid presenting a synthetic example as an observed productivity result.

Diagnose failures in the right layer

When a search returns no record, first check whether the intended source is in the collection and whether the query wording appears in it. When the expected record is present but ranks poorly, examine competing records and the query. When the correct record is retrieved but the answer is wrong, examine interpretation and generation. These are different problems. Rewriting a prompt will not restore a document that was never indexed or decide which obsolete version should be trusted.

When the utility reports an input error, inspect the CSV headers, empty fields and repeated identifiers. A missing document identity can make later evidence impossible to trace even if the text itself is searchable. The supplied script rejects these basic input problems rather than quietly loading them. That is a limited data-quality check, not a validation that every sentence in the body is accurate. Keep the technical and editorial checks distinct.

Maintain one clear next action for each failure. A vocabulary mismatch may lead to a query alternative or a clearer title. Missing authority may lead to a document-owner review. An unsupported question may lead to a new procedure request. A malformed CSV may lead to an export correction. The baseline becomes useful when it directs work to the right person instead of turning every difficulty into a request for a larger model or more generated text.

Frequently asked questions

Is the downloadable utility an AI assistant?

No. It is a local keyword retrieval demonstration using SQLite full-text search. It returns source records and does not generate answers, call a model or perform business actions. That limited behavior is intentional because it provides a baseline for later comparisons. If the sources cannot be found or their authority is unclear, those problems should be visible before an answer-writing system is added.

Does running it keep every copy of my data private?

The script itself makes no network calls and uses an in-memory database, but the input CSV and any redirected output remain wherever you save them. Your computer, backups and other software have their own handling behavior. Use the supplied fictional file for the exercise. A real-data pilot needs an approved data path and access design; local execution alone does not settle every privacy or security question.

Why might a complete question return no result?

The demonstration treats the query as a literal phrase for full-text matching. A natural-language question may contain words that do not occur together in a record. Try a shorter phrase while retaining the original question in your evaluation sheet. The difference illustrates a limitation of this baseline. It does not establish that a semantic system will always find the correct source or that the collection lacks relevant information.

Can a high-ranked result be an obsolete document?

Yes. Ranking concerns how the indexed text matches the query, not whether your organization approves that version. The conflicting-version exercise makes that distinction explicit. A real system needs a documented rule for current sources and an appropriate implementation of that rule. Do not expect an answer generator to resolve conflicting authority simply because both documents look plausible or one appears first in the result list.

What should I do if FTS5 is unavailable?

Use a Python and SQLite installation that supports the required extension through your normal approved setup process. The script reports that requirement clearly. Do not describe a different search algorithm as the same tested example without changing its documentation and tests. You can still complete the question-and-source worksheet manually while deciding whether the technical demonstration is useful for your environment.

When is outside implementation help relevant?

When the problem extends to connectors, permissions, source authority, ongoing updates or integration with a real team workflow, the tiny demonstration has reached its boundary. Bring the collection description, evaluation questions, observed failures and proposed user group to the responsible technical and operations owners. Elixir's AI workflow implementation service is a relevant contextual resource for that broader process discussion, while this guide remains an educational baseline you can run with fictional data.