AI Gateway — Fallback Routing

What happens when your primary LLM provider goes down? Watch AI Gateway automatically route to the fallback. Zero downtime for your users.

The problem: Last year OpenAI had a 4-hour outage. Anthropic had outages too. Every LLM provider has incidents. If your app calls these APIs directly, your app goes down with them. AI Gateway sits between your code and any AI provider — when the primary fails or rate-limits, traffic flows to a fallback automatically. Your users never see the outage.
Ready. Click a button above to run.

Primary

First choice
@cf/meta/llama-3.1-8b-instruct-fast
Waiting...
↓
if fails

Fallback

Backup if primary fails
@cf/meta/llama-3.2-3b-instruct
Waiting...

Routing Decision

Final response delivered to user

AI Gateway — Per-User Rate Limiting via Dynamic Routing

A different problem: what if the primary is up, but one user is hammering it? Watch AI Gateway block abusive traffic per user so your bill isn't held hostage by a single bad actor — without a code change.

How it works: A Dynamic Route is a flow you define in the AI Gateway dashboard. This one has a Rate Limit node keyed to metadata.userId at 3 requests / minute. Under the limit → 70B model responds normally. Over the limit → gateway blocks the request before it reaches the provider (zero LLM cost incurred). The application code just calls dynamic/cost-tiered as if it were a single model name. The routing and blocking live in the platform.

Three outcomes to watch for in the log:
  1. Fresh call — 70B (allowed), latency in the 500-3000ms range, real per-call cost.
  2. Rate-limit block — BLOCKED, latency under 150ms, cost $0. Gateway rejected the request before it reached the provider.
  3. Cache hit — CACHED, latency under 50ms, cost $0. AI Gateway returned a stored response for an identical prompt it has seen before. No provider call, no tokens billed. The latency drop from 2000ms to 20ms is how you know caching is on.

The Fallback branch can also route to a cheaper model instead of blocking — the platform supports both patterns. This demo blocks because the Beta editor currently has issues persisting a fallback model. When that lands, we’ll show the graceful downgrade variant here too.
Ready. Click a button above to run.
(no calls yet)