Apple researchers have developed a method to significantly reduce the computational cost of Retrieval-Augmented Generation (RAG) by replacing traditional geometric search with small, learned neural networks. By training models to predict the 'support function' of a database, the system can route queries to the correct data clusters more accurately than standard centroid-based methods, achieving up to 8x faster retrieval for high-recall results.

The Limitation of Geometric Search

Maximum Inner Product Search (MIPS) is the engine behind recommendation systems and RAG, where a system must find the best-matching 'key' vector for a given query. As databases grow to millions of keys and dimensions reach into the thousands, exhaustive comparison becomes computationally prohibitive.

Current approximate methods often use clustering to group keys around centroids. However, these methods assume queries will resemble the keys. In RAG, this is often false: user queries are typically short, while the keys are long, curated pieces of text, leading to routing errors where the best match is skipped because it sits far from its cluster's center.

Diagram showing how geometric search can fail to find the nearest neighbor due to centroid-based partitioning.
Figure 1 illustrates how traditional geometric approaches can miss the nearest neighbor because they rely on partitioning space based on distances to centroids.machinelearning.apple.com

This image explains the fundamental limitation of geometric search where queries might be routed to the wrong cluster because they are far from the centroid.

“The SupportNet model can answer a more useful question than a simple comparison to the centroid can: “How good is the best key inside this cluster for this query?”

— Faster Retrieval for Better, More Efficient RAG

SupportNet and KeyNet Architectures

The researchers identified that the MIPS value function is a 'support function' in convex analysis. They developed two specialized models to exploit this: SupportNet and KeyNet. SupportNet is an input-convex neural network trained to approximate the support function of each cluster, answering the question: 'How good is the best key inside this cluster for this query?'

Comparison of centroid-based assignment versus support function-based assignment for cluster routing.
Figure 2 demonstrates how relying on distances to centroids can be misleading, whereas using a cluster's support function allows for more accurate assignment.machinelearning.apple.com

This visual supports the transition from centroid-based routing to the SupportNet approach which considers the best key within a cluster.

Visualization of SupportNets approximating support functions to assign queries to clusters based on best matches.
Figure 3 shows how SupportNets approximate support functions to assign queries based on the best match with any key in a cluster, rather than just the centroid.machinelearning.apple.com

This image illustrates the mechanism of SupportNet as described in the architecture section.

KeyNet models learn to directly output the best-matching key vector. By using a consistency loss derived from Euler’s theorem, these models remain geometrically grounded even without explicit convexity constraints. Both models are trained on the specific distribution of queries that actually matter, such as those found in the Natural Questions benchmark.

Performance Gains and Trade-offs

In testing, the SupportNet model hit 95% routing recall about 1.9x faster than traditional centroid routing. When using the largest model, the system achieved 97% recall in 1.74 ms, demonstrating that the small neural networks can outperform geometric baselines even when accounting for their own inference overhead.

Chart showing routing performance recall versus latency for SupportNets of various sizes.
Figure 4 displays the routing performance and latency trade-offs for SupportNets of different sizes and scan fractions.machinelearning.apple.com

This chart provides the empirical evidence for the performance gains and trade-offs discussed in the final section of the article.

SupportNet Performance vs. Centroid Routing: Centroid Routing 3.36 ms; Small SupportNet 1.74 ms
SupportNet Performance vs. Centroid Routingmachinelearning.apple.com

Chart drawn by HN Digest from figures reported by machinelearning.apple.com; values are shown as written in the source.

KeyNet provides a different advantage by acting as a query rewriter. By transforming a raw query into a vector that is much closer to the target key, it can reduce the number of keys an index needs to scan to reach 99% recall by up to 11x. While these models require an upfront cost to train on a fixed database, they provide a more efficient path for production systems with stable query distributions.

Comparison of top-3 hits for a raw query versus a KeyNet-mapped query for the question 'What is Matchbox Twenty?'.
Figure 7 shows how KeyNet improves retrieval by mapping a raw query to a vector that more closely aligns with the database's curated keys.machinelearning.apple.com

This example demonstrates the 'query rewriter' capability of the KeyNet model mentioned in the performance section.