How do you keep your AI's answers up to date?
Written lesson · about 3 minutes to read
The short answer
An AI tool that answers from your documents is only as current as those documents, so give each one an owner and a review date, remove old versions, and make sure the tool re-reads changes. Then run the same test questions every month, because a stale answer can go unnoticed until a customer is quoted last year's price.
Three things to remember
- 01The tool is only as current as its documents, and someone has to own that job.
- 02Take old versions out of the folder the tool reads, instead of adding new files beside them.
- 03Run the same test questions every month, and make it easy for staff to flag a wrong answer.
Why answers go stale
How current the tool is depends on how it reads your files. If it was handed a folder in March, it knows March. If it is connected to the live folder and re-reads on a schedule, it knows the last time that ran. If it reads the document at the moment of the question, it knows today. Ask your vendor which one you have.
The first kind is common and cheap, and it goes wrong quietly. Nothing looks broken and nobody sees an error message. A customer is simply told last year's price in a confident voice, and only then does someone find out.
Owners, review dates and old versions
Give every document a named owner and a review date. The price book gets reviewed whenever supplier prices change. A procedure gets reviewed whenever the way you do the job changes. One person also owns the whole set: what is in the folder, what is archived, and when the tool re-reads it.
Take old versions out instead of leaving them beside the new one. If the old policy and the new policy both sit in the folder, the search finds both, and the tool may pick one or blend them. Move old versions to an archive the tool does not read.
Check the tool re-reads changes
The tool cannot see a changed file until it re-reads it. Some products do that within minutes, some overnight, and some only when someone presses a button. Find out which, and match it to how fast your facts move. Prices that change weekly need at least a nightly re-read. A staff contact list that changes twice a year can wait.
Test it monthly and let staff flag mistakes
Keep a short list of test questions with the correct answers, and put the same questions to the tool once a month. Include questions whose answers recently changed. Set the tool to show the source document and its date with each answer, so staff can see how old the source is.
Give staff an easy way to say an answer is wrong: a button in the tool, a shared channel or a word with the owner. Read a small sample of real answers each month for the ones that slipped through.
Where to start this week
Name the one person who owns the sources. Move every superseded price list and policy into an archive folder. Write ten test questions with correct answers and today's date on each, and put a monthly reminder in the owner's calendar.
Then ask how your tool re-reads changes, and set up the route for staff to flag a wrong answer. The next time a price changes, replace the file the same day and run the test questions.
More in part 5
- 1What data does your AI actually need?
- 2Garbage in, garbage out: cleaning up before you automate
- 3How do you give AI your price book, manuals and SOPs?Written
- 4What is a knowledge base, and does your business need one first?Written
- 5What is RAG?
- 6What is a vector?
- 7What are embeddings?
- 8What is a vector database?
- 9What is semantic search?
- 10What is chunking?
- 11What is re-ranking?
- 12What is AI memory?
- 13What is fine-tuning?
- 14Fine-tuning vs. RAG vs. prompting: which one do you need?
- 15How do you keep your AI's answers up to date?WrittenYou are here
Next step
Where does this fit your business?
Six quick questions about your business, then your AI plan: what to start with, what a person still checks, and the lessons that go with it. Or bring it straight to a free thirty-minute call.