Respan Gateway functions as a translation layer for artificial intelligence applications. Developers often face the challenge of integrating multiple LLM providers, each of which may use different API structures, request formats, and authentication methods. Respan addresses this by providing a single endpoint where a client can call any model, and the gateway handles the heavy lifting of translating that request into the specific format required by the provider. For example, a client using an OpenAI-compatible SDK can request a Claude model, and the gateway will automatically route and reformat the request to Anthropic's API.
The gateway acts as a translation layer, allowing a single endpoint to serve multiple providers.
The system is built in Rust and operates as a single binary that includes a web UI for monitoring. Its core functionality is modularized into a plugin architecture, where features like routing, pricing, and tool execution are treated as distinct components. The gateway supports a wide array of capabilities beyond simple chat completions, including real-time voice (translating between OpenAI Realtime and Gemini Live), image generation, speech-to-text, and video jobs. It also integrates with the Model Context Protocol (MCP), allowing it to front MCP servers and run their tools directly within chat requests for clients that do not have native MCP support.
The system is built in Rust and features a modular plugin architecture for various AI capabilities.
One of the primary design choices in Respan is its reliance on a plugin-based system for its core features. While this allows for high extensibility—enabling users to add custom guardrails, PII masking, or specialized routing logic—it also introduces specific trade-offs regarding complexity and security. Because the gateway is designed to be highly customizable, users can build their own plugins using a public Rust API. However, this modularity means that the stability and security of the gateway are tied to the integrity of these plugins. The project provides several pre-built plugins for common tasks, such as 'hedged races' (sending requests to multiple providers to find the fastest response) and 'fallback' mechanisms to ensure high availability when a specific provider experiences downtime.
Despite its broad compatibility, the gateway's behavior is governed by a configuration file that defines client keys, providers, and specific routes. For instance, a user can define a 'smart' route that automatically selects between different models based on the request. The gateway also provides observability features, including a live request log and per-call pricing, which helps developers track costs across different vendors in a centralized location. By abstracting the underlying provider details, Respan aims to simplify the 'LLMops' workflow, allowing developers to focus on application logic rather than the nuances of individual vendor APIs.
While the gateway simplifies multi-model integration, it also introduces a new point of failure and a potential bottleneck in the infrastructure. Because all requests pass through the Respan binary, the gateway must be scaled and managed effectively to maintain low latency. Furthermore, while the plugin system offers flexibility, it requires developers to manage the lifecycle and dependencies of those plugins. The project currently supports a vast array of vendors, including OpenAI, Anthropic, Gemini, Groq, and DeepSeek, making it a comprehensive tool for those seeking to avoid vendor lock-in by maintaining a single, consistent interface for their AI stack.