Decision Model RAG Reranker

Decision Model RAG Reranker

Semantic similarity finds related text, but related text does not always answer the question. This workflow adds a Decision Model after vector retrieval to judge each chunk against the actual question. It keeps the strongest evidence and asks an LLM to answer from that evidence alone.

What this workflow does

1. Question receives a question about your documents.
2. RetrieveCandidates searches your vector store for up to eight chunks. Built-in Cohere reranking is disabled because the Decision node performs this step.
3. CandidateLoop sends each chunk, together with the original question, to RelevanceJudge.
4. The Decision Model returns a Score from 0 to 3: unrelated, tangential, useful partial evidence, or a direct answer.
5. CollectEvidence preserves each chunk's text, ID, metadata, vector similarity, and new relevance score.
6. SelectEvidence drops scores below 2, sorts the rest by decision relevance, and keeps the top three.
7. HasEvidence runs GroundedAnswer only when evidence survived. Otherwise NoEvidence asks the user to refine the question or add documents.

Setup

Select an existing vector store on RetrieveCandidates and set its database type to match. The default is Qdrant. Ingest your documents first; this template only searches them. Select a Decision Model credential on RelevanceJudge and an LLM credential on GroundedAnswer. Change the starter model if needed.

The default question is about a refund policy. For a useful first test, ingest a direct refund-policy passage, a tangential pricing passage, and an unrelated deployment note. Inspect how vector similarity and decision relevance differ.

Reading the reranked result

The selected chunks are available at $SelectEvidence.chunks. Each item has id, text, metadata, vector_score, and relevance. The answer includes chunk IDs as citations. The original vector score is retained for inspection but does not determine the final order.

Score is probability-weighted, so values can fall between the four rubric levels. The cutoff of 2 and the top-three limit are examples to tune against your own retrieval evaluation set.

Empty results and cost

An empty search or a set of irrelevant chunks reaches NoEvidence without generating an answer. A successful search makes one decision call per retrieved chunk and, when evidence passes the cutoff, one LLM call. The loop is sequential and collects results explicitly in a run variable. This is a transparent reranking example, not a claim that it outperforms a dedicated reranker on every dataset.