RAG for AI agents, and how we do it at Qopper

A language model knows nothing about your business. It knows how to talk. Ask it how many days a customer has to return an order and it will answer with a number that sounds reasonable, almost always thirty, because thirty is what it has read most often. It isn't lying to you: it's finishing your sentence.

That is the whole distance between an agent that answers well and one that answers truthfully. And what closes it has a name: RAG.

What RAG is

RAG stands for retrieval-augmented generation. In terms of what actually happens: before answering, the agent searches your business documents for the few paragraphs that matter, reads them, and writes the answer with them in front of it.

It's what a new person on your team does. They don't memorize the manual on day one: they learn where the manual is, and when someone asks about the return window they open it to the returns page. The quality of their answer doesn't depend on their memory; it depends on finding the right page.

RAG doesn't make the agent smarter. It makes it stop guessing.

It's worth saying what it isn't, too. Pasting your policies into the agent's instructions looks similar, but it isn't the same and it breaks quickly: that text isn't searched, can't be cited, doesn't say which document it came from, and travels in full with every message even when the question has nothing to do with it. With two pages it holds up. With an airline's real manual, it doesn't.

How it works, step by step

Under the hood it's six moves. The first four happen once, when you upload a document; the last two, every time someone writes.

1. 1It's split. The document is cut into section-sized chunks. Whole, it's useless: if the answer is two sentences, handing the agent forty pages buries them.
2. 2It's turned into numbers. Each chunk becomes a list of numbers that represents its meaning. Two texts that mean the same thing in different words land close together. That's what lets “it arrived broken” find the “defective product” section.
3. 3It's also indexed as text. In parallel, the same chunk is indexed word by word. It's the half almost everyone skips, and it's the one that finds a policy number, a reference or a proper name.
4. 4The question arrives. What the customer writes is converted the same way as the chunks: into the same kind of numbers, so they can be compared.
5. 5It searches and picks. Candidates come back from both searches and the best ones stay. Not all of them: the few that really matter.
6. 6It answers with the source in front of it. Those paragraphs go into the answer along with the document and page they came from. That's why the agent can say where it got the deadline it just quoted.

Where textbook RAG breaks

Setting this up in an afternoon is easy, which is why there are so many. The hard part is making it hold up in a real conversation. These are the four places it breaks, and we've seen all four break.

• It only searches by similarityMeaning-based search is great with phrases and terrible with identifiers. A customer who types their policy number wants that document, not one that talks about policies in general.
• It searches with the last messageHalf of what gets written in a thread can't be searched on its own. “And how long does that take?”, “the other one too?”, “with tax?”. Searching that returns nothing, the paragraph with the answer never arrives, and the model has to improvise.
• When it finds nothing, it returns something anywayIt's the most expensive failure of all. If there's no match and “the top chunks” get handed over, the agent cites them as if they applied. Giving it irrelevant context is worse than giving it none.
• It confuses retrieving with knowingA retrieved paragraph says what the business wrote. It doesn't say what state this person's order is in right now. Those are two different questions, answered in different places.

How we do it at Qopper

Our agent doesn't just answer: it checks your systems, makes changes within the limits you set, and verifies before saying it's done. That raises the bar for retrieval, because here a wrong number isn't a so-so answer, it's a promise your business will have to keep or take back.

• Two searches, alwaysEvery question is searched by meaning and word by word at the same time. The first finds what the customer meant; the second, the exact SKU, policy number and proper name.
• Agreement wins, not the scoreThe two lists are merged rewarding overlap: a paragraph that comes second in both beats one that comes first in only one. Two different searches pointing at the same thing is the best signal there is that it's what was being looked for.
• A short question is completed with the previous oneWhen the message is too short to carry its own subject, it's searched together with the customer's last message. Length decides, not another model call: asking an AI whether a sentence depends on the previous one costs a full round trip per turn to get right what counting characters already gets right.
• Eight paragraphs per turnNot three, not forty. Three leave out the exception written right below the rule; forty dilute the answer in noise.
• Each agent searches only its ownDocuments are connected agent by agent. The sales agent doesn't quote the returns manual, and an agent with no documents connected says it doesn't know instead of answering from someone else's manual.
• No match, no contextIf the search finds nothing relevant, it isn't padded with whatever was closest. The agent says it has no record of that, which is the right answer and the one nobody builds because it doesn't look impressive.
• Every paragraph arrives with its sourceDocument and page travel attached to the text all the way to the answer. That's what turns “it's fifteen days” into “it's fifteen days, and it's in your returns policy”.
• And each company's data never crosses overEvery search filters by the company that owns the document before looking at anything else. It's not a check at the end: it's the first condition of the query.

Retrieving isn't knowing

RAG solves one half: what your business has written down. The other half is what's happening right now in your systems, and a document doesn't help there. When the agent looks up an order, what comes back is the real state at that moment, tagged with the system it came from and the time it was read.

What matters isn't what comes back, it's what's missing. If the carrier hasn't assigned a delivery date yet, the agent doesn't see a gap: it sees an object without that line, and fills it with whatever sounds reasonable. That's how an agent ends up saying “it arrives Thursday” when nobody ever said Thursday. That's why absences are named: “no delivery date on record” is as useful as the date, and it's what stops the agent from making one up.

And before telling a customer something is done, it's checked against the system that says it is.

Knowledge goes stale, and it does so silently

This is the failure that worries us most, because nothing breaks. You change the return window on your site from thirty days to fifteen, nobody re-imports the page, and the agent keeps promising thirty for months. No error, no alert, and every answer is a commitment you won't keep.

That's why we watch your pages section by section, not as a whole: a fingerprint of the full page changes when the year in the footer changes, alerts every time, and within two weeks nobody looks at it. And not every change weighs the same. A section that says “we're happy to help” can change with no consequences; one that says “fifteen days”, “30%” or “$50,000” is exactly where the numbers the agent promises come from. Those get flagged separately.

How to tell if yours works

If you're evaluating an agent, yours or someone else's, four questions separate what works from what demos well.

• Where did this come from?Ask it for the source of a specific number. If it can't name the document, it didn't retrieve it: it already knew it, which is another way of saying it made it up.
• What does it do when it doesn't know?Ask it something that isn't written anywhere. The right answer is that it has no record of it. Anything else, however well written, is the problem.
• Does it hold up on the second question?Ask something long, then just “and how long does that take?”. That shows whether it searches the conversation or just the last line.
• Does it know the data changed?Change a number on your site and see how long it takes to notice, or whether someone has to remember to tell it.

None of this is exotic. It's the craft of making an agent say only what it can back up, which is the only thing that makes it worth putting in front of your customers.