Choose an LLM gateway by the controls it enforces when something goes wrong, not the number of models it lists. For a platform team handling private workloads, put private-model isolation first, then provider failover, per-team budgets, exportable audit evidence, and configuration ownership. A gateway that routes requests but lets callers bypass those controls hasn’t solved the production problem.
Start With the Gateway’s Enforcement Boundary
A gateway governs only the traffic you force through it. Before comparing features, draw the path from your application to the gateway, the model, and the systems an agent can call afterward.
Separate the data plane, which handles requests, from the control plane, which manages configuration. Mark where provider credentials, prompts, cached responses, logs, and routing decisions live. A private inference server doesn’t keep data private if an external classifier receives the prompt first.
Give each application or agent a distinct Non-Human Identity (NHI), meaning an identity for software rather than a person. Bind that identity to an owning team and allowed destinations; don’t let an untrusted request header decide who pays or which policy applies.
NIST’s Zero Trust Architecture rejects implicit trust based on network location and calls for authentication and authorization before a resource session is established. For a private model, translate that principle into a reachability test, not just a successful login.
Use a staging environment with synthetic prompts, independent team identities, approved public and private destinations, and a controllable failing backend. The acceptance conditions below are proposed procurement requirements, not reported benchmark results or claims that every product already meets them.
The Acceptance Checklist, in Priority Order
Treat isolation as a gate: a high score elsewhere doesn’t compensate for a forbidden destination receiving a prompt. The order below prioritizes sensitive workloads; a public-content application can put availability first without dropping the other checks.
| Criterion | Why it matters | Repeatable acceptance test | Pass condition and evidence |
|---|---|---|---|
| Private-model isolation | A private route is useless if callers or fallback rules bypass it. | Send a private-only request, disable its backend, then retry from an unauthorized identity. | Only approved private destinations receive content; the unauthorized identity can’t reach the protected service. Save destination and denial logs. |
| Provider-routing failure | A working primary provider hides retry and compatibility problems. | Inject rate-limit errors, server errors, timeouts, and an interrupted stream. | Only approved fallback targets are tried, within your application deadline; partial output isn’t silently spliced. Save the attempt trace. |
| Per-team token and spend enforcement | Usage reports don’t stop an agent from consuming a budget. | Exhaust a test team’s quota while another team continues; repeat with parallel requests. | The exhausted team is blocked according to the agreed cap semantics, while the other team still works. Save counter and rejection evidence. |
| Audit export | A dashboard isn’t evidence your incident team can retain and query. | Export successful, denied, retried, and configuration-change events to your own destination. | You can reconstruct the caller, policy, destination, usage, outcome, and administrator change without vendor interpretation. Save raw exported records. |
| Configuration ownership | Controls drift when dashboard edits and application overrides escape review. | Apply a reviewed change, attempt an unauthorized override, then restore the previous state. | Runtime policy matches the approved state, unauthorized changes fail, and rollback is reproducible. Save the diff and administrative audit events. |
Test Each Control Against Failure
Run the same workload and failure sequence against every candidate. Record the product version, edition, deployment topology, enabled policies, and client configuration so another engineer can reproduce the result.
Private Models: Prove There Is No Bypass
Private-model access must control both destination choice and the network path to inference. An API permission doesn’t prove an unauthorized workload can’t connect directly to the backend.
- Assign a synthetic request to a private-only policy and confirm the actual destination at the backend.
- Disable the private backend and inspect every attempted fallback, including classifiers and embedding services used to choose a route.
- Try an explicit public model name, a client-supplied routing configuration, and direct backend access from an unauthorized workload.
- Revoke the authorized identity and repeat the request, checking both new connections and any existing session.
Evidence to request: the destination allowlist, identity-to-policy mapping, backend connection records, and the documented revocation behavior. Require a diagram that includes telemetry exporters and caches, not just the inference request.
Red flags: private-only requests silently fall back to a public provider; the protected private backend remains reachable by unauthorized workloads outside the gateway; or the vendor equates encryption with authorization. For sensitive traffic, specify fail-closed behavior: reject the request when no approved destination is available.
NetFoundry’s secure gateway deployment recipe explains why its example runs prompt embeddings locally before choosing a private or public route. The useful lesson isn’t that semantic classification guarantees privacy; it’s that the classifier’s own data path belongs in the review.
Provider Failure: Test the Attempt Chain, Not the Happy Path
Failover passes only when the backup is permitted, compatible, and fast enough for your application. Switching to a healthy model doesn’t help if its output breaks your tool schema or violates a regional restriction.
- Force the primary backend to return a rate-limit error, then a server error, then a timeout.
- Record which target receives each attempt and whether the client and gateway both retry.
- Interrupt a streaming response after output has started, then test client cancellation.
- Stop a gateway instance separately from stopping a provider to expose shared infrastructure dependencies.
Evidence to request: retry triggers, timeout settings, maximum attempts, route restrictions, and the complete request trace. Include tool-call arguments, structured output, and error handling in compatibility checks rather than testing plain text alone.
Kong documents configurable failover criteria and retry settings. Portkey documents prioritized fallback targets and customizable status-code triggers, so ask each vendor to show the configuration that produced its result.
Red flags: the application receives fragments from different models as though they were one response, every application retries an already-retried call, or the gateway’s own failure has no recovery path. Set the total deadline before tuning retries; don’t accept an availability feature that consumes the caller’s entire timeout.
Team Budgets: Separate Tokens, Money, and Throughput
A token quota, a spending cap, and a rate limit answer different questions. Tokens measure model input and output, spending caps constrain attributed cost, and rate limits constrain throughput over a time window.
Portkey’s limits documentation explicitly distinguishes accumulated budgets and token limits from rolling rate limits. Require that distinction in the demo, especially when different models have different token prices.
- Give separate teams independent identities and deliberately small test quotas.
- Exhaust one quota, then repeat through a different allowed model and fallback route.
- Run parallel requests near the boundary and inspect how in-flight usage is counted.
- Try changing the team label, creating another key, and calling the provider directly.
Evidence to request: the counter’s scope, reset rules, source of token counts, pricing inputs, and documented handling of concurrent or unfinished requests. Ask whether input and output tokens, cached requests, failed attempts, and retries count toward each limit.
Red flags: only an alert fires, usage updates after the next billing cycle, or the caller can pick an arbitrary cost center. If usage is reconciled after completion, agree on an overshoot policy and test it; don’t describe that behavior as a strict pre-request ceiling.
For a small team, a key-scoped cap can be sufficient when key issuance is controlled. A shared enterprise gateway needs aggregation across keys and replicas, separate environments, and a clear rule for which budget wins when team and project limits overlap.
Audit Export: Reconstruct an Incident Outside the UI
Auditability means your incident team can recover the event sequence from exported records. Model-request logs and administrative audit logs serve different purposes, and you need both.
Generate an allowed request, a denied private-model attempt, a fallback, a budget rejection, and a routing-policy change. Export the resulting records into the system your security team actually uses, then ask an engineer who didn’t watch the demo to reconstruct what happened.
Request these fields where applicable:
- Workload identity, owning team, environment, and request correlation ID.
- Requested model, resolved model, actual provider, and fallback attempts.
- Policy decision, rejection reason, token usage, and accounting status.
- Administrator identity, changed object, prior state, new state, and timestamp.
Kong’s AI audit reference describes structured AI analytics. Portkey’s export documentation identifies JSONL export, an Enterprise-plan requirement, and a Data Service requirement for self-hosted Enterprise customers.
Evidence to request: a real export sample, schema documentation, retention and deletion rules, access controls, and behavior when the export destination fails. Test whether necessary event metadata remains available when prompt and response content logging is disabled.
Red flags: screenshots offered instead of records, request IDs lost across retries, or a claim that usage metrics alone prove who changed a policy. Don’t collect full prompts by default merely because the gateway can log them.
Configuration: Keep Policy Out of the Caller’s Hands
Configuration ownership means you can review, apply, inspect, and reverse policy without giving applications administrative authority. Self-hosting helps with runtime placement, but it doesn’t automatically provide change approval or rollback.
- Store a proposed route and budget change in your approved configuration system, excluding secrets.
- Preview and apply the change, then compare the live state with the reviewed state.
- Attempt a dashboard edit with a read-only account and a request-level override with an application identity.
- Restore the previous state, rotate a provider credential, and check what happens during control-plane disconnection.
Evidence to request: role permissions, an API or declarative configuration format, change history, drift detection, and the recovery procedure. Document who owns gateway patches, identity lifecycle, provider credentials, and emergency exceptions.
Kong’s kongctl guide documents declarative configuration with a preview-and-apply workflow. Portkey’s Configs reference documents request-level configurations and their precedence over client defaults, making override permissions a specific acceptance question rather than an abstract lock-in concern.
Red flags: exporting configuration also exports credentials, application callers can weaken mandatory policy, or a rollback requires the vendor to rebuild your environment. A file in source control isn’t proof of control until the runtime follows it.
What to Verify With Kong and Portkey
Use each product’s documented strengths to choose where to start the demonstration. Neither a feature name nor a successful setup proves the failure tests pass.
Kong: Start With Existing API Operations

Kong AI Gateway is a practical starting point when your platform team already operates Kong-based API controls. Its documentation covers provider routing, team-level token budgets, AI policies, observability, and programmatic administration.
Ask for a complete run of the routing, budget, and configuration tests on the edition you will deploy. Kong’s main documentation distinguishes Konnect management from on-prem configuration paths, so don’t transfer a capability demonstrated in one topology to another without checking support.
For private-model access, require the same unauthorized-connection test as every other candidate. An established API operating model is valuable, but it isn’t evidence that your private backend has no bypass path.
Portkey: Inspect Budget Scope and Export Entitlement

Portkey’s homepage now identifies the product as Prisma AIRS AI Gateway. Its documented fallback configuration and separate usage-control scopes make it a useful candidate when model operations and team accounting drive the decision.
Run the budget test at the workspace boundary, not just against an individual key. Then verify that the deployment and purchased entitlement include the export mechanism your incident team needs; seeing logs in a UI isn’t the same as being entitled to export them.
Portkey’s hybrid architecture places the data plane in customer infrastructure while Portkey hosts the control plane. Ask for the exact configuration and telemetry flows, then test disconnection and recovery instead of assuming hybrid means fully independent operation.
NetFoundry Against the Same Acceptance Criteria

NetFoundry’s AI Enclave is a relevant starting point when private reachability is the constraint. Its published architecture describes cryptographic workload identities and authorization before a routable path to protected resources exists, the authenticate-before-connect principle behind Identity-First Reachability.
Keep the platform’s published capabilities separate from the open-source gateway repository. The repository documents provider routing, private overlay connections, virtual keys, metrics, and file-based configuration; it isn’t proof that every commercial governance control ships in the standalone binary.
- Private-model isolation: documented architecture, deployment test required. The AI Enclave page describes identity-authorized reachability and outbound-only connections. Prove the deployed model and gateway have no direct unauthorized path, including any optional public access mode.
- Provider failure: documented, scope needs validation. The repository describes health checks and passive failover across inference endpoints. Demonstrate the specific cross-provider fallback and interrupted-stream behavior your application requires rather than assuming endpoint failover covers both.
- Team budgets: platform capability stated, standalone coverage unproven. The AI Enclave page describes budgets and cost attribution by agent, team, and project. The repository README doesn’t establish equivalent team-cap enforcement, so request the exact enforcing component and run the parallel-request test.
- Audit export: acceptance gap until demonstrated. The platform describes identity-based records, while the repository documents metrics and optional request-body logging. Neither source establishes the export format, retention commitment, or administrative change trail required by this checklist.
- Configuration ownership: file-based control documented, operational controls still yours to verify. The repository provides YAML configuration and environment-variable references for provider credentials. Require a demonstration of approval, drift detection, secret handling, and rollback for the product you buy.
NetFoundry’s real limitation in this evaluation is that documented private reachability doesn’t settle budget semantics or audit-export requirements. If your main need is governed public-model consumption and your existing gateway passes the tests, private networking alone isn’t a reason to replace it.
For a private deployment discussion, bring the destination map and rejection traces to NetFoundry’s team. Ask them to close the unresolved acceptance items, not repeat the architecture presentation.
FAQ
Do gateways add latency?
A gateway adds another processing step, and guardrails, classification, caching, and retries change the end-to-end result. Measure time to first token and completion time with your own prompts, both with and without the intended policies enabled. Include slow backends and streaming requests so the measurement reflects your application rather than an empty proxy.
Is an Open-Source Gateway Enough for Production?
An open-source license gives you code access, not an operating model. Before production, assign owners for patching, high availability, secrets, policy changes, and incident evidence, then verify which controls require a separate edition or service. Budget for the dependencies and on-call work needed to keep the deployment running.
What does OpenAI-compatible actually guarantee?
OpenAI-compatible describes an interface, not identical behavior across every provider and endpoint. Test the exact client features you use, including streaming, tool calls, structured responses, cancellation, and usage reporting. Keep a representative request-and-response fixture for each required capability so upgrades can be checked against the same contract.
Should you enable semantic caching immediately?
Enable semantic caching only after testing tenant isolation, invalidation, and whether similar prompts are allowed to share an answer. Keep it disabled for personalized or sensitive workloads until those rules are explicit and verified. Include a cache-hit case in your audit and accounting tests rather than treating it as a provider call.
Can an LLM API Gateway Also Secure Agent Tools?
Model routing doesn’t prove that a later tool invocation is authorized. Evaluate tool access separately, using MCP gateway policy checks when agents connect through the Model Context Protocol, the protocol used to expose tools and data to AI applications. Correlate model requests with tool events without assuming that either gateway sees traffic that bypasses it.
What changes as usage grows across teams?
Shared gateways need consistent counters across replicas, controlled key issuance, tenant-separated logs, and a recovery plan for shared dependencies. Repeat the acceptance suite at expected concurrency and during an instance failure before expanding access. Assign an owner to cross-team exceptions so emergency changes don’t become permanent routes around policy.
Sources & References
- NIST Zero Trust Architecture: identity before resource sessions.
- Kong AI Gateway documentation: controls and deployment distinctions.
- Kong load balancing: retries and failover configuration.
- Kong AI audit reference: structured AI event records.
- Kong configuration management: declarative change workflow.
- Portkey: current product naming.
- Portkey limits: budget and throughput distinctions.
- Portkey fallbacks: target order and triggers.
- Portkey logs export: export format and entitlements.
- Portkey Configs: configuration precedence.
- Portkey hybrid architecture: data and control-plane placement.
- NetFoundry AI Enclave: platform identity and governance model.
- NetFoundry deployment recipe: local classification data path.
- Open-source gateway repository: runtime configuration and documented limits.