Context Engine benchmarks

Overview

Three sets of measurements. Each one states its data, its questions and its limits. Each one compares the Context Engine with itself or with a plain baseline: no other product appears in any of them.

Permissions under narrow access

The Context Engine applies permissions during the search. The common alternative is to search first and filter afterwards. That loses results: the search spends its limit on passages the person may not see, and what is left after the filter is short or empty, with no error. This benchmark measures how much.

On a 100,000-document corpus with synthetic team-shaped permissions, a person who may see 10% or 1% of the documents gets exactly the top ten that a corpus holding only their documents returns. The same search filtered afterwards returns nothing for 64% and 99% of searches. Fetching ten times more before the filter recovers 0.42 and 0.02 of the results, and fifty times more recovers 0.64 and 0.08.

What was compared

• Permissions during search. The Context Engine’s search, run as the person.
• Filter after, same limit. The same search with no permissions, then the top 10 filtered to what the person may see.
• Filter after, 10x and 50x fetch. The top 100 or the top 500 with no permissions, filtered, cut to 10. Fifty times is the deepest fetch measured.
• The reference. The same search, with the same settings, on a copy that holds only the documents the person may see, searched exactly. Recall@10 is the share of that copy’s top ten a method returned. “Empty” is the share of searches where a method returned nothing although the copy had results.

Results

100,646 passages, clustered (team-shaped) permissions, the first 200 questions of the dataset. The 95% interval, a percentile bootstrap over questions, is under each recall figure.

Where every search agreed with the reference the bootstrap interval has no width. The bound that does is the rule of three: with 200 searches and no disagreement, up to 1.5% of searches could still differ. “The same top ten” on this page means the same set of ten results, not the same order.

Random against clustered permissions

The same number of visible documents, drawn at random instead of as topic clusters. With permissions applied during the search the layout makes no difference: recall against the visible-only corpus is 1.000 either way. Filtering after the search depends on it. Each cell is recall@10 with the share of searches returned empty under it.

Hybrid search, filtered after the search

At the same limit the two layouts lose about the same recall. What clustering changes there is how the losses are spread: more searches come back with nothing at all. Fetching more is where the layout decides the outcome. A benchmark that assigns permissions at random and over-fetches will report that filtering after the search is nearly safe. The longer argument is in Your ACL benchmark is measuring the easy case.

Conditions

• Synthetic permissions. No real permission data was used. For the clustered layout, 40 seed documents are drawn at random, every document joins its most similar seed, and whole clusters are taken until the person sees 10% or 1% of the corpus. Real permissions follow topic less cleanly.
• Agreement, not relevance. A recall of 1.000 here means the person got what a deployment holding only their documents would return. It does not say those results answer the question.
• Single-group people, no public documents. Each person holds one group and every document carries a permission.
• Data. BEIR HotpotQA, 100,000 documents (the judged documents of 500 questions, the rest sampled with seed 0), embedded with BAAI/bge-small-en-v1.5 at 384 dimensions, default settings. The questions are the dataset’s questions in dataset order and were not chosen to match the visible topics.

Retrieval quality

Hybrid search at the default settings against plain vector search with the same embeddings, on five public datasets, the first 300 questions of each, with BAAI/bge-small-en-v1.5 embeddings. Every comparison is paired question by question on the same data, and the interval is a 95% bootstrap of the per-question differences.

nDCG@10, with the 95% interval of the paired difference under each delta. “Tie” means the interval spans zero. Recall@10, vector only then hybrid: MuSiQue 0.558 / 0.556; 2WikiMultihopQA 0.752 / 0.745; MultiHop-RAG 0.766 / 0.790; ArguAna 0.833 / 0.817; CQADupStack English 0.581 / 0.584.

Corpus sizes, documents and passages: MuSiQue 11,656 / 12,211; 2WikiMultihopQA 6,119 / 6,817; MultiHop-RAG 609 / 6,976; ArguAna 8,674 / 12,142; CQADupStack English 10,000 of 40,221 / 10,924.

Context Graph

Each dataset was uploaded once in graph mode and searched over that one upload, with and without the Context Graph. Every configuration answered the same 300 questions. Graph mode is a choice per upload, and nothing here changes a default.

Searching the graph raised nDCG@10 over hybrid search on two of the four datasets (2WikiMultihopQA by +0.044 and GraphRAG-Bench novel by +0.046, each with a 95% interval above zero), made no measurable difference on MuSiQue, and lowered it slightly on MultiHop-RAG (-0.016).

Hybrid search with the graph, minus hybrid search alone, on the same 300 questions, with the 95% interval under each figure.

nDCG@10 on the capped corpus of each dataset. “Documents” is the cap out of the dataset’s full size.

• The graph widens what is found more than it sharpens the top of the list. Recall@10 went up on all four datasets, beyond its interval on the same two. The rank of the first relevant document did not improve in the same way.
• Model cost of a graph upload. Building the graph cost between $2.48 and $3.26 of model spend per 1,000 passages with the extraction model used here, and a graph search makes no model call. The model was Gemini 2.5 Flash with thinking switched off, at its list price at the time of the run ($0.30 per million input tokens, $2.50 per million output tokens): $3.23 on MuSiQue, $3.26 on 2WikiMultihopQA, $2.49 on GraphRAG-Bench novel, $2.48 on MultiHop-RAG.
• The corpora are capped, and the cap was set by cost. A cap keeps every judged document of the 300 questions and fills the rest with distractors drawn with seed 0. With fewer distractors every configuration scores higher than it would on the full corpus, so these absolute numbers are not comparable with the retrieval-quality section above.

What these benchmarks do not show

• Hybrid search does not beat vector search here. On five public datasets with a small embedding model, hybrid search is ahead of plain vector search on one, level on one and behind on three. A stronger embedding model changes the balance. A corpus of part numbers, error codes, names or misspellings may need keyword matching more than these datasets do.
• The Context Graph does not always help. Two of four datasets, all built from multi-hop questions, where a graph is expected to matter most. Single-hop questions and full-size corpora were not measured.
• Not answer quality. Every figure is retrieval only. No answer was generated or graded.
• Not real permissions. The permission layouts are generated, each person holds a single group, and every document carries a permission. People in several groups, documents visible to everyone, and narrowing by source or document were not measured.
• Nothing beyond the measured scale. The permission run is 100,000 documents. Fetching more than fifty times before filtering is not measured.
• One dataset, one embedding model for the permission benchmark. Longer documents, another model or another dimension move the numbers for filtering after the search.
• Questions are not aimed at the visible topics. A person who searches only for what their own team wrote will find filtering after the search less harmful than these numbers. A person who searches for anything else will not.
• Not latency. No figure on this page is a claim about search time.
• Not a comparison with any other product. Both sides of every comparison are the Context Engine’s own search or a plain baseline.