Jeeves builds on the Qwen3.5-9B language model, applying LoRA with rank 16 on all projections and adding a pointer head to score answer choices. A block‑4 diffusion drafter, inspired by Orthrus but adapted to Qwen3.5’s Gated DeltaNet layers, generates speculative tokens to speed up reasoning. The model is first fine‑tuned with supervised learning on 19,126 questions from public datasets and synthetic policy data, then further optimized with CISPO reinforcement learning for up to 402 steps. It supports yes/no (noul), multiple‑choice (choice) and rating (score) questions through a Jev‑compatible API, allowing a single request to contain mixed question types.
The article describes Jeeves as a 9B parameter decision model that adds a reasoning phase to improve accuracy, as shown in the project repository.
On the developer’s test split (2,962 items) Jeeves achieves 0.889 accuracy when thinking is enabled, outperforming the reported Kev‑9B score of 0.822 and the Jev baseline of 0.857. On JevBench’s public tiers it scores 0.935, compared to Jev’s 0.866 and the unpublished Kev‑9B figure (represented by Kev‑8B at 0.715*). Without thinking the same checkpoint scores 0.804, showing that the reasoning chain adds about 0.085 absolute accuracy. Latency measurements on an H100 GPU with FP8 precision show a median of 3.3 seconds per request with full chains and about 0.3 seconds without thinking; the 90th‑percentile latency reaches 17.1 seconds. On an M4 Pro, one question thinks at roughly 20 tokens per second, and a request without thinking takes 0.3 to 0.5 seconds. FP8 quantization reduces the model’s memory footprint from 21 GB to 11.5 GB on bf16 weights, with only negligible changes in output on development questions.
The model exhibits known trade‑offs. On knowledge‑focused benchmarks Jeves trails the base Jev: MMLU scores 0.793 versus Jev’s 0.900, and MMLU‑Pro 0.739 versus 0.840. The authors note that thinking latency grows at the tail, with p90 latency of 17 seconds on a single H100, which may be unsuitable for latency‑critical applications. Additionally, no explicit language consistency reward was included during training, so the generated reasoning chains are not guaranteed to be interpretable. The project advises using max_think to truncate reasoning length (e.g., 768 tokens drops median latency to 2.0 seconds) or applying a nothink_threshold to skip thinking when confidence is high, and highlights that the diffusion drafter’s default block‑4 setting offers a balance between throughput and quality when processing multiple questions in a batch.