API platform

Azure API Management

APIM is the front door for every API you expose — REST, GraphQL, gRPC, WebSocket, SOAP and, increasingly, LLM and agent endpoints. It decouples consumers from backends with a programmable gateway, a management plane, and a developer portal.

How it works: core architecture

APIM has three building blocks that scale and deploy independently:

flowchart LR
    subgraph Consumers
        C1[Web / mobile apps]
        C2[Partner systems]
        C3[Internal services]
        C4[AI agents / copilots]
    end
    subgraph APIM[API Management]
        GW[Gateway
policies · auth · cache · rate limits]
        DP[Developer portal]
        MP[Management plane]
    end
    subgraph Backends
        B1[App Service / AKS /
Container Apps]
        B2[Azure Functions]
        B3[Logic Apps]
        B4[Azure OpenAI /
AI Foundry]
        B5[On-prem & SaaS APIs]
    end
    C1 --> GW
    C2 --> GW
    C3 --> GW
    C4 --> GW
    GW --> B1
    GW --> B2
    GW --> B3
    GW --> B4
    GW --> B5
    MP -.configures.-> GW
    DP -.onboards.-> Consumers
      

The policy pipeline

Policies are XML-defined statements executed sequentially in four sections. They can apply at global, workspace, product, API, or operation scope, and compose across scopes via <base />.

sequenceDiagram
    participant Client
    participant GW as APIM Gateway
    participant BE as Backend
    Client->>GW: Request
    Note over GW: INBOUND
validate-jwt · rate-limit · quota ·
ip-filter · rewrite-uri · set-header ·
cache-lookup · mock-response
    GW->>BE: Forward (BACKEND section:
retry · forward-request · timeout)
    BE-->>GW: Response
    Note over GW: OUTBOUND
set-body · redact fields ·
cache-store · CORS headers
    GW-->>Client: Response
    Note over GW: ON-ERROR
custom error shaping · logging ·
trace to App Insights
      

Policies you will actually use

Example: JWT validation + spike protection

<inbound>
  <base />
  <validate-azure-ad-token tenant-id="{{tenant-id}}">
    <audiences><audience>api://my-api</audience></audiences>
    <required-claims>
      <claim name="roles" match="any"><value>Orders.Read</value></claim>
    </required-claims>
  </validate-azure-ad-token>
  <rate-limit-by-key calls="100" renewal-period="60"
      counter-key="@(context.Subscription.Id)" />
  <set-header name="X-Correlation-Id" exists-action="skip">
    <value>@(Guid.NewGuid().ToString())</value>
  </set-header>
</inbound>

AI Gateway: fronting Azure OpenAI & AI Foundry

This is the fastest-moving part of APIM. The same gateway that protects REST APIs now acts as a GenAI gateway in front of Azure OpenAI, Azure AI Foundry model deployments, and third-party/self-hosted LLMs — giving you one governed entry point for every model and agent in the company.

• One-click import of Azure OpenAI and AI Foundry endpoints as APIs (OpenAI-compatible or Azure AI Model Inference API), with keyless managed-identity auth to the backend.
• Token governance — per-team token limits (llm-token-limit) and token metrics (llm-emit-token-metric) for chargeback across cost centers.
• Model load balancing & failover — backend pools spread traffic across regional deployments (round-robin, weighted, priority) and the circuit breaker skips throttled (429) deployments; PTU-first with PAYG spillover is the canonical setup.
• Semantic caching — embedding-based cache so repeated/similar prompts don't hit the model at all.
• Content safety — inspect prompts centrally before they reach any model.
• MCP support — expose existing APIM-managed APIs as MCP servers (tools) and gateway remote MCP servers, so agents in AI Foundry, GitHub Copilot or any MCP client consume governed tools with APIM policies (auth, rate limits, observability) in front.
• Foundry integration — AI Foundry can front its model deployments with an AI Gateway; agents built in Foundry Agent Service call tools published through APIM, keeping enterprise APIs governed while agents act on them.

flowchart TB
    subgraph Clients
        A1[Apps & copilots]
        A2[AI Foundry Agents]
        A3[MCP clients
GitHub Copilot · Claude · VS Code]
    end
    subgraph APIM[APIM · AI Gateway]
        P1[llm-token-limit
llm-emit-token-metric]
        P2[llm-semantic-cache]
        P3[llm-content-safety]
        P4[Backend pool
load balancing + circuit breaker]
        MCP[MCP server façade
REST APIs exposed as tools]
    end
    subgraph Models[Model backends]
        M1[Azure OpenAI
PTU deployment]
        M2[Azure OpenAI
PAYG deployment]
        M3[AI Foundry
model deployments
DeepSeek · Llama · Mistral]
    end
    subgraph Tools[Enterprise APIs]
        T1[Orders API]
        T2[Inventory API]
    end
    A1 --> P1
    A2 --> P1
    A3 --> MCP
    A2 --> MCP
    P1 --> P2 --> P3 --> P4
    P4 -->|priority 1| M1
    P4 -->|priority 2 spillover| M2
    P4 --> M3
    MCP --> T1
    MCP --> T2
      

Tiers & deployment options

flowchart LR
    subgraph Azure
        MPlane[APIM management plane
+ developer portal]
        CGW[Cloud gateway]
    end
    subgraph OnPrem[On-premises / other cloud / edge]
        SHGW[Self-hosted gateway
container in K8s]
        LB[Local backends]
    end
    Client1[Cloud clients] --> CGW
    Client2[Local clients] --> SHGW
    SHGW --> LB
    SHGW -.config sync & telemetry.-> MPlane
    MPlane -.configures.-> CGW
      

Detailed use cases

Best practices

• Adopt APIOps: APIs and policies in Git, deployed via CI/CD (APIOps toolkit / Bicep), never hand-edited in production.
• Use managed identity from gateway to backends; never store backend keys in policy when avoidable — use named values + Key Vault when you must.
• Version APIs explicitly (path or header versioning) and use revisions for non-breaking changes.
• Always return a Retry-After on 429s from token limits; clients behave better.
• Front every LLM endpoint with the AI Gateway policies from day one — retrofitting cost controls is painful.
• Send gateway logs/traces to Application Insights + Log Analytics; alert on 4xx/5xx ratios and token consumption anomalies.
• Prefer v2 tiers for new instances; Premium classic only when you need multi-region or full VNet injection today.

Key references

• APIM documentation
• Policy reference index
• GenAI gateway capabilities
• Expose APIs as MCP servers
• AI-Gateway labs (Azure Samples)
• GenAI gateway reference architecture