Making company knowledge findable

The situation

Every established business accumulates knowledge that is technically available and practically unreachable. Contracts, procedures, past proposals, support history, specifications and years of email sit in systems people can access and cannot search usefully.

The practical consequence is that people ask each other. The same questions route to the same two or three individuals, who become a bottleneck and a concentration risk, and a great deal of senior time is spent answering things that were written down years ago.

Retrieval-augmented generation genuinely solves this, and the naive version of it fails in specific and predictable ways — surfacing superseded documents with confidence, leaking material people were not entitled to see, and inventing answers when retrieval finds nothing.

How it shows up

• The same questions get asked internally every week.
• Two people are the only ones who know how something works.
• Support agents search a wiki nobody has updated in a year.
• New staff take months to become productive.
• Proposals are rewritten from scratch because finding the previous one is harder.
• A previous chatbot attempt made things up and was quietly retired.

Symptom, cause and change

The most expensive mistake in this situation is treating a symptom as a diagnosis. These are the three columns kept apart.

Why it happens

No curation

Everything was indexed, including drafts, duplicates and superseded versions, so retrieval surfaces them with equal confidence.

Semantic search alone

Pure vector search handles exact terms — part numbers, clause references — poorly, which is precisely what people search for.

Permissions applied after retrieval

Filtering the answer rather than the candidate set means confidential content has already been read and can leak through paraphrase.

No refusal behaviour

A system that must always answer will invent one when retrieval fails, which is worse than returning nothing.

Stale index

Indexed once and never refreshed, so it confidently quotes procedures that were replaced eighteen months ago.

How we approach it

1. Curate the corpus

   Decide what is authoritative, what is superseded and what should never be indexed. This needs someone who knows the material, it cannot be delegated to us, and it determines the ceiling on everything that follows.

2. Collect the real questions

   From support tickets, internal chat and the people who currently get asked. These become the evaluation set, which is what makes retrieval tuning measurable rather than impressionistic.

3. Engineer retrieval and measure it separately

   Hybrid keyword and semantic search with reranking, chunked to document structure. Retrieval quality is scored independently of answer quality, because conflating them hides which half is failing.

4. Enforce permissions at retrieval

   Entitlement filters applied to the candidate set using your existing identity system, so a user’s results never contain material they could not already open.

5. Ground, cite and refuse

   Answers constrained to retrieved passages with citations back to source, and an explicit refusal path when nothing relevant is found. Refusal is tested deliberately.

6. Automate re-indexing

   Pipelines that pick up changes from the source systems, so the knowledge base reflects current documents without anyone remembering to refresh it.

What changes

Answers with sources

Every response cites the passage it used, so people verify rather than trust blindly.

The bottleneck eases

Recurring questions get answered by the system, returning time to whoever was fielding them.

Confidential material stays confidential

Permissions enforced at retrieval rather than filtered from the output.

It admits ignorance

Explicit refusal when retrieval finds nothing, tested as a behaviour rather than hoped for.

Knowledge stays current

Automatic re-indexing, so superseded procedures stop being quoted with authority.

New staff ramp faster

A searchable corpus with citations is a materially better onboarding tool than a shared drive.

Where it goes wrong

The dominant failure is indexing everything. A shared drive contains drafts, duplicates, superseded versions and a great deal of material nobody should be reading, and retrieval surfaces all of it with the same confidence. Curation is most of the work and it is the part clients most want to skip.

The second is misdiagnosing retrieval failures as model failures. If the right passage is never returned, no model produces the right answer, and teams spend weeks adjusting prompts to fix a search problem.

The third is permissions applied to the output rather than the retrieval. By the time the model has read a confidential document, paraphrase can leak it regardless of what filter sits after. Entitlements belong in the query.

A fourth is having no refusal path. Clients frequently find a refusing assistant disappointing in a demonstration and reliable in production, and the systems that always answer are the ones that eventually say something embarrassing.

A fifth is treating it as a one-off build. Corpora go stale, models change underneath you, and a system evaluated once was evaluated for one day. Re-indexing and re-evaluation are ongoing or the thing degrades invisibly.

Finally, retention of query logs. They record what employees asked, which is sensitive in ways that surprise people. The retention period is a policy decision that should be made deliberately rather than left at a vendor default.

A further failure is launching to everyone at once. A narrow first audience — one team, one document set — surfaces the corpus problems while they are still cheap to fix, and a full rollout that begins badly is very difficult to recover reputationally inside an organisation.

Finally, nobody owns the corpus after launch. Documents are superseded, procedures change, and without a named owner for what is authoritative the index drifts back toward the shared-drive problem it was built to solve.

What else you could do instead

A retrieval system is not the only way to solve this, and sometimes a simpler answer captures most of the value.

Fix the search you already have

Many document systems have usable search that has never been configured. Proper metadata, consistent naming and a curated structure can resolve a large share of the problem at almost no cost.

Write down the twenty questions

If the same questions recur, documenting them properly in one findable place solves it without any technology. Unglamorous and frequently sufficient.

Curate without building

A great deal of the benefit comes from deciding what is authoritative and archiving what is not. That work is valuable independently and is a prerequisite for anything else.

Buy an off-the-shelf assistant

Where the corpus is small and non-sensitive and permissions are simple, a commercial product may cover it. We would say so rather than building.

How we would know it worked

We measure retrieval and generation separately and report both against the real question set with its size stated. Retrieval is scored on whether the correct passage appears in the returned candidates; generation on whether the answer is correct and grounded in it.

Alongside accuracy we track refusal rate and human override rate. A system that never refuses is inventing; one that refuses constantly is not retrieving. Override rate — how often people correct it — is the most honest signal available and frequently disagrees with the accuracy figure.

The organisational measure is whether the bottleneck eased, which we baseline by asking the two or three people who currently field the questions to record volume for a fortnight before launch. Without that baseline, any later claim about time saved is an estimate rather than a measurement, and we would label it as one.

How long it takes and what it costs

A first working system over a curated corpus is typically six to twelve weeks, with the variation driven almost entirely by document variety and by how long curation takes on your side rather than by engineering.

Corpus curation is the long pole and it is largely your work. Where an organisation can dedicate someone to it, the project moves quickly; where it is fitted around other duties, it is the reason these projects run to four months instead of two.

Cost is quoted per phase after we have seen a sample of the corpus. Running cost depends on query volume and on whether inference is local or hosted, and we model it against expected usage before you commit.

Estimates are labelled as estimates. Timelines here are planning ranges from comparable work, not commitments, and not measured client outcomes. We quote against a defined scope after a discovery call.

Services involved

Other situations

• Replacing spreadsheets with a real system
• Cutting cost per qualified lead
• Launching in a new European market
• Automating quote to invoice
• Recovering from a failed migration
• Inheriting undocumented software
• Passing a customer security review
• Merging systems after an acquisition
• Opening a second location