The concept of a 'decision hierarchy' proposes a structured approach to utilizing artificial intelligence by treating different models as layers in a memory-like hierarchy. In this framework, the lowest level (L-cache) consists of hand-written rules and exact semantic matches of past answers. The middle layer (RAM) is occupied by specialized decision models, such as Jev, which provide well-defined outputs like scores or probabilities. The highest level (disk) is the Large Language Model (LLM), while the final fallback is human intervention. Each level is designed to handle what it is most confident about and pass 'misses' down to the more computationally expensive or complex layer below.
This figure visualizes the multi-layered approach where different models handle specific confidence levels.
To determine when to move from a cheaper decision model to a more expensive LLM, the author introduces a mathematical framework based on 'Chow's rule.' This rule calculates a break-even point by weighing the cost of an error against the cost of the next level's processing fee. For example, if a decision model is only 60% confident, the system must decide if the risk of a wrong answer (weighted by the cost of that error) is higher than the fee and expected error rate of the LLM. When errors are highly expensive, the system prioritizes accuracy, making the LLM's fee less relevant. Conversely, when errors are cheap, the processing fee becomes the primary factor in deciding whether to escalate the query.
The framework further refines this by accounting for variable error costs across different categories. Because a mistake in a legal routing might be significantly more costly than a mistake in a technical routing, the rule is adjusted to incorporate category-specific overhead. The resulting model suggests that the optimal path for any given query is a hyperbola-shaped trade-off between the probability of a correct answer and the economic cost of the intelligence required to produce it. This approach moves beyond simple confidence thresholds, instead treating AI deployment as a series of economic decisions where the goal is to minimize the total cost of both processing and mistakes.