At a Glance:
Portkey AI Gateway is an open-source, self-hosted API gateway that provides unified routing to 250+ LLMs from 45+ providers with built-in fallbacks, retries, load balancing, caching, and guardrails.
Overview:
Portkey AI Gateway is a lightweight, open-source API gateway purpose-built to route requests to over 250 language, vision, audio, and image models from more than 45 providers through a single fast and friendly API. It is designed for developers who need a reliable and secure integration layer for large language models without managing separate provider SDKs. The gateway can be deployed locally via Docker, Node.js, or Cloudflare Workers, and it also supports private enterprise deployments on AWS, Azure, GCP, and Kubernetes. In addition to core routing, it offers a centralized MCP Gateway for managing model context protocol servers with authentication, access control, and observability.
Key Decision Points:
Self-hosted deployment: Can be deployed locally using Docker, Node.js, or Cloudflare Workers, or privately on AWS, Azure, GCP, OpenShift, and Kubernetes.
API compatibility: Exposes an OpenAI-compatible API signature, allowing drop-in replacement with existing OpenAI SDKs and frameworks like LangChain and CrewAI.
Centralized MCP gateway: Provides a separate control plane for managing MCP servers with authentication, role-based access control, and per-call logging across an organization.
Enterprise features gated: Advanced capabilities like prompt template management, provider optimization, semantic caching, and full observability are only available in the hosted and enterprise versions.
Multi-modal and realtime support: Routes requests for vision, audio (TTS and speech-to-text), image generation models, and OpenAI realtime APIs through integrated WebSocket connections.
Core Features:
Fallbacks and automatic retries: Automatically fail over to another provider or model on request errors, with configurable retry logic and exponential backoff.
Load balancing: Distribute requests across multiple API keys or providers using weighted routing for higher availability.
Guardrails: Validate LLM inputs and outputs against 40+ pre-built checks for security and accuracy, with support for custom and partner guardrails.
Smart caching: Serve repeat queries from a cache to reduce latency and cost, supporting both simple and semantic caching in hosted and enterprise versions.
Secure key management: Use your own API keys or generate virtual keys dynamically; supports role-based access control and PII redaction in enterprise deployments.
Use Cases:
Developers building multi-provider AI applications: Use the gateway to switch between LLM providers without modifying application code, leveraging fallbacks and load balancing for reliability.
Teams integrating agent frameworks: Connect agent frameworks like Autogen, CrewAI, or LangChain to multiple LLM backends through a single API endpoint with centralized logging.
Organizations managing MCP server access: Deploy the MCP Gateway to enforce authentication, control tool-level access, and monitor all MCP calls from clients like Claude Desktop and VS Code.
Open-Source Alternative Value:
As an open-source gateway, Portkey AI Gateway allows developers to self-host a central routing layer for multiple LLM providers, avoiding reliance on any single vendor's API. The codebase is available for local deployment, inspection, and customization, providing a lightweight route to multi-provider reliability features like fallbacks, retries, and caching. Organizations with stricter deployment requirements can move to private enterprise deployments on their own cloud infrastructure, while the community edition remains available for smaller-scale projects and experimentation.
