Traefik Labs Advances LLM and MCP Runtime Governance with Composable Safety Pipeline, Multi-Provider Resilience, and Token-Level Cost Controls

Audio By Carbonatix

SANTA CLARA, Calif.--(BUSINESS WIRE)--Mar 16, 2026--

NVIDIA GTC - Traefik Labs today announced new capabilities that extend Traefik Hub's Triple Gate architecture (API Gateway, AI Gateway, and MCP Gateway) with deeper runtime governance across the full AI workflow: a composable, multi-vendor safety pipeline with parallel guard execution, multi-provider failover routing, token-level cost controls, graceful error handling for agent-aware enforcement, IBM Granite Guardian integration, and a new Regex Guard capability that lets organizations write their own custom guards.

These capabilities address a growing gap. Enterprises moving to autonomous agents face fragmented governance: CSP-native tools are locked to a single cloud, SaaS gateways route traffic through third-party infrastructure, and a growing number of "LLM proxies" and "MCP proxies" typically focus on a single or narrow layer of traffic. An LLM proxy sees the model interaction but not the agent actions that follow. An MCP proxy sees tool calls but not the LLM conversation that triggered them. Even as some begin to expand their scope, few offer composable multi-vendor safety, cost controls, resilience, and agent authorization in a unified architecture.

"You can't govern the full AI workflow by looking at one layer at a time," said Sudeep Goswami, CEO at Traefik Labs. "Enterprises need an infrastructure-native approach that enforces safety, cost control, resilience, and agent authorization from a unified and integrated platform they own and operate, across any environment. That's what Traefik's Triple Gate architecture makes possible, and today's release takes it further."

A Composable Safety Pipeline That Runs in Parallel

Traefik Hub's AI Gateway now supports a multi-vendor safety pipeline, allowing organizations to choose from multiple guardrail providers and combine them. Total enforcement time equals the slowest guard, not the sum.

The pipeline spans four tiers:

• Regex Guard (NEW): A framework for organizations to write their own guards using regex-based pattern matching, at sub-millisecond speed with zero external dependencies. Teams define rules to block or mask content based on patterns they know best: Social Security numbers, credit card formats, API keys, or any pattern specific to their business. For well-understood patterns like PII, there's no reason to pay the latency and cost of an AI model when a regex match is faster, cheaper, and deterministic.

• Content Guard (Microsoft Presidio): Global PII detection and masking with statistical NLP-based entity recognition, supporting both built-in and custom entity patterns.

• LLM Guard with NVIDIA NIMs: GPU-accelerated jailbreak detection, content safety across 22+ categories, and topic control provides semantic intelligence for threats that deterministic patterns can't catch.

• LLM Guard with IBM Granite Guardian (NEW): IBM's open-source safety models provide harm detection, jailbreak detection, topic control, hallucination detection, and RAG quality assessment, capabilities that other guard providers don't yet offer.

Parallel Guard Execution (NEW): LLM-based guards, the heavyweight tiers that can take multiple seconds to execute, now run in parallel rather than in series. Guards are classified as critical (failure blocks and cancels) or optional (failure is logged). Multiple NVIDIA NIMs (jailbreak, content safety, topic control) and IBM Granite guards can all execute simultaneously, so total enforcement time equals the slowest guard, not the sum. Traefik will continue integrating leading third-party providers as the guardrail ecosystem matures.

Operational Controls: Resilience, Cost Control, and Agent-Aware Enforcement

• Failover Router (NEW): Automatic failover across LLM providers and models via circuit breaker chain. The router switches both providers and models, enabling cost-optimized degradation while all safety policies remain enforced. Organizations can mix OpenAI, Anthropic, self-hosted models, and NVIDIA NIMs in a single failover chain, something not possible when governance is locked to a single CSP.

• Token Rate Limiting and Quota Management (NEW): Tracks input, output, and total tokens independently, with rate limiting for spikes and quotas for hard budget caps. Per-user, per-team, per-endpoint, or per-API-key tracking via JWT claims. Proactive token estimation blocks abusive requests before the prompt reaches the LLM, rather than enforcing limits reactively after resources are consumed.

• Graceful Error Handling (NEW): When a guardrail blocks a request, traditional gateways return an HTTP 403 that breaks agent control flow and crashes multi-step workflows. Traefik Hub's guardrails can now be configured to return structured, schema-compliant refusal responses (HTTP 200 with a refusal message) that agents and applications can process gracefully. Agents continue operating, middleware chains stay intact, and users see conversational refusals instead of technical errors. This is what makes the runtime governance agent-aware: enforcement that works with autonomous and agentic workflows rather than breaking them.

Why This Matters Now

AI agents are no longer experimental. Gartner predicts 40% of enterprise apps will feature AI agents by the end of 2026, up from less than 5% in 2025. The Model Context Protocol has grown to over 10,000 published servers. NVIDIA unveiled NemoClaw at GTC 2026 this week, bringing enterprise-grade agent orchestration with application-level governance to the NVIDIA stack. OpenClaw gives users autonomous agents that coordinate across 50+ services. Salesforce has over 12,000 Agentforce implementations in production.

Yet infrastructure-layer governance has not kept pace. Agent platforms are adding application-level controls (RBAC, audit logging, signed skills) but these operate inside the agent runtime. If the runtime is compromised, so are the guardrails. Only 14.4% of enterprises get full security approval before deploying agents (Okta Enterprise AI Agent Security Survey), and 80% of organizations report risky agent behaviors, including unauthorized system access (HelpNet Security, February 2026). What's missing is enforcement at the traffic layer: independent of the agent platform, below the application, governing every API call, LLM interaction, and tool invocation as it crosses the network.

Traefik Hub's Triple Gate is the only unified infrastructure-layer approach that governs LLM content safety, cost, and resilience alongside agent authorization through Tools/Tasks/Transactions-Based Access Control (TBAC for MCP Gateway), on infrastructure you operate, from public cloud to air-gapped environments. It works with any agent platform (NemoClaw, LangChain, CrewAI, or custom) because it governs the traffic, not the runtime.

Traefik's infrastructure-native approach is already gaining traction with neoclouds, service providers, and enterprises building GPU-accelerated AI infrastructure. Organizations that have standardized on Traefik for application networking can add the full AI Gateway and MCP Gateway capabilities through a single in-place upgrade, with no re-architecture, no traffic migration, and no additional proxies in the data path.

Availability

Traefik Hub v3.20 is now available as an Early Access release, with general availability planned for late April 2026.

For a detailed technical walkthrough of these new capabilities, see the accompanying blog post: From Regex to GPU: Building a Multi-Vendor AI Safety Pipeline.

To try these capabilities, sign up for Early Access.

About Traefik Labs

Traefik Labs is the company behind Traefik Proxy, the world's most widely deployed cloud-native application proxy with over 3.4 billion downloads and 62k+ GitHub stars, and Traefik Hub, the modern API management platform for API, LLM and MCP governance. For more information, visit traefik.io.

View source version on businesswire.com:https://www.businesswire.com/news/home/20260316864823/en/

CONTACT: Press Contact

Dylan Rodgers

[email protected]

KEYWORD: UNITED STATES NORTH AMERICA CALIFORNIA

INDUSTRY KEYWORD: DATA MANAGEMENT SECURITY TECHNOLOGY SOFTWARE NETWORKS ARTIFICIAL INTELLIGENCE INTERNET

SOURCE: Traefik Labs

Copyright Business Wire 2026.

PUB: 03/16/2026 02:30 PM/DISC: 03/16/2026 02:30 PM

http://www.businesswire.com/news/home/20260316864823/en