AI-Gateway

# 🧪 Model Context Protocol (MCP) Labs with Azure API Management

[![Open Source Love](https://firstcontributions.github.io/open-source-badges/badges/open-source-v1/open-source.svg)](https://github.com/firstcontributions/open-source-badges)

## What's New ✨

A new workshop experience is now available, allowing you to explore AI features like token limits, monitoring, and load balancing via the Azure Portal.

[Go to the Workshops](https://aka.ms/ai-gateway/workshop)

<div>
  <img src="./images/workshop.png" alt="workshop" width="600">
</div>

Explore the following labs to experiment with new and exciting features:

*   ➕ **Realtime API (Audio and Text) with Azure OpenAI 🔥** experiments with the [**AOAI Realtime**](labs/realtime-audio/realtime-audio.ipynb)
*   ➕ **Realtime API (Audio and Text) with Azure OpenAI + MCP tools 🔥** experiments with the [**AOAI Realtime + MCP**](labs/realtime-mcp-agents/realtime-mcp-agents.ipynb)
*   ➕ **Model Context Protocol (MCP) ⚙️** experiments with the [**client authorization flow**](labs/mcp-client-authorization/mcp-client-authorization.ipynb)
*   ➕ The [**FinOps Framework**](labs/finops-framework/finops-framework.ipynb) lab to manage AI budgets effectively 💰
*   ➕ **Agentic ✨** experiments with [**Model Context Protocol (MCP)**](labs/model-context-protocol/model-context-protocol.ipynb).
*   ➕ **Agentic ✨** experiments with [**OpenAI Agents SDK**](labs/openai-agents/openai-agents.ipynb).
*   ➕ **Agentic ✨** experiments with [**AI Agent Service**](labs/ai-agent-service/ai-agent-service.ipynb) from [Azure AI Foundry](https://azure.microsoft.com/en-us/products/ai-foundry).
*   ➕ The [**AI Foundry Deepseek**](labs/ai-foundry-deepseek/ai-foundry-deepseek.ipynb) lab with Deepseek R1 model from [Azure AI Foundry](https://azure.microsoft.com/en-us/products/ai-foundry).
*   ➕ The [**Zero-to-Production**](labs/zero-to-production/zero-to-production.ipynb) lab with an iterative policy exploration to fine-tune the optimal production configuration.
*   ➕ The [**Terraform flavor of backend pool load balancing**](labs/backend-pool-load-balancing-tf/backend-pool-load-balancing-tf.ipynb) lab.
*   ➕ The [**AI Foundry SDK**](labs/ai-foundry-sdk/ai-foundry-sdk.ipynb) lab.
*   ➕ The [**Content filtering**](labs/content-filtering/content-filtering.ipynb) and [**Prompt shielding**](labs/content-filtering/prompt-shielding.ipynb) labs.
*   ➕ The [**Model routing**](labs/model-routing/model-routing.ipynb) lab with OpenAI model based routing.
*   ➕ The [**Prompt flow**](labs/prompt-flow/prompt-flow.ipynb) lab to try the [Azure AI Studio Prompt Flow](https://learn.microsoft.com/azure/ai-studio/how-to/prompt-flow) with Azure API Management.
*   ➕ `priority` and `weight` parameters to the [**Backend pool load balancing**](labs/backend-pool-load-balancing/backend-pool-load-balancing.ipynb) lab.
*   ➕ The [**Streaming**](streaming.ipynb) tool to test OpenAI streaming with Azure API Management.
*   ➕ The [**Tracing**](tools/tracing.ipynb) tool to debug and troubleshoot OpenAI APIs using [Azure API Management tracing capability](https://learn.microsoft.com/azure/api-management/api-management-howto-api-inspector).
*   ➕ Image processing to the [**GPT-4o inferencing**](labs/GPT-4o-inferencing/GPT-4o-inferencing.ipynb) lab.
*   ➕ The [**Function calling**](labs/function-calling/function-calling.ipynb) lab with a sample API on Azure Functions.

## Table of Contents

1.  [🧠 GenAI Gateway](#-genai-gateway)
2.  [🧪 Labs with AI Agents](#-labs-with-ai-agents)
3.  [🧪 Labs with the Inference API](#-labs-with-the-inference-api)
4.  [🧪 Labs based on Azure OpenAI](#-labs-based-on-azure-openai)
5.  [🚀 Getting Started](#-getting-started)
6.  [⛵ Roll-out to Production](#-roll-out-to-production)
7.  [🔨 Supporting Tools](#-supporting-tools)
8.  [🏛️ Well-Architected Framework](#-well-architected-framework)
9.  [🎒 Show and Tell](#-show-and-tell)
10. [🥇 Other Resources](#-other-resources)

The rapid advancement of AI necessitates experimentation-driven strategies for organizations to stay competitive. As AI transforms various sectors, continuous innovation is crucial for businesses aiming to fully leverage its potential.

**AI services** are primarily accessed through **APIs**, highlighting the importance of a robust and efficient API management strategy. This strategy is vital for maintaining control and governance over the consumption of **AI services**.

With the expanding capabilities of **AI services** and their integration with **APIs**, there's a growing need for a comprehensive **AI Gateway** pattern, which extends the core principles of API management. This aims to accelerate the experimentation of advanced use cases and foster further innovation. The well-architected principles of the **AI Gateway** provide a framework for confidently deploying **Intelligent Apps** into production.

## 🧠 GenAI Gateway

![AI-Gateway flow](images/ai-gateway.gif)

This repository explores the **AI Gateway** pattern through a series of experimental labs. The [GenAI Gateway capabilities](https://techcommunity.microsoft.com/t5/azure-integration-services-blog/introducing-genai-gateway-capabilities-in-azure-api-management/ba-p/4146525) of [Azure API Management](https://learn.microsoft.com/azure/api-management/api-management-key-concepts) are central to these labs, managing AI services APIs with security, reliability, performance, operational efficiency, and cost controls. The primary focus is on [Azure OpenAI](https://learn.microsoft.com/azure/ai-services/openai/overview), which serves as the standard reference for Large Language Models (LLM). However, the principles and design patterns can be applied to any LLM.

Recognizing the prevalence of Python in AI and the experimental capabilities of Jupyter notebooks, the labs are structured around Jupyter notebooks with step-by-step instructions, Python scripts, [Bicep](https://learn.microsoft.com/azure/azure-resource-manager/bicep/overview?tabs=bicep) files, and [Azure API Management policies](https://learn.microsoft.com/azure/api-management/api-management-howto-policies).

## 🧪 Labs with AI Agents

### [**🧪 MCP Client Authorization**](labs/mcp-client-authorization/mcp-client-authorization.ipynb)

This lab provides a playground to experiment with the [Model Context Protocol](https://modelcontextprotocol.io/) using the [client authorization flow](https://modelcontextprotocol.io/specification/2025-03-26/basic/authorization#2-10-third-party-authorization-flow). In this flow, Azure API Management acts as both an OAuth client connecting to [Microsoft Entra ID](https://learn.microsoft.com/en-us/entra/architecture/auth-oauth2) and as an OAuth authorization server for the MCP client ([MCP inspector](https://modelcontextprotocol.io/docs/tools/inspector) in this lab).

[<img src="images/mcp-client-authorization-small.gif" alt="flow" style="width: 437px; display: inline-block;" data-target="animated-image.originalImage">](labs/mcp-client-authorization/mcp-client-authorization.ipynb)

[🦾 Bicep](labs/mcp-client-authorization/main.bicep) ➕ [⚙️ Policy](labs/mcp-client-authorization/src/weather/apim-api/policy.xml) ➕ [🧾 Notebook](labs/mcp-client-authorization/mcp-client-authorization.ipynb)

**Key MCP Functionality Explored:**

*   **Client Authorization Flow:** Understand how to implement the third-party authorization flow within MCP using Azure API Management.
*   **OAuth Client and Server Roles:**  Learn how Azure API Management can function as both an OAuth client and server in the MCP ecosystem.
*   **Integration with Microsoft Entra ID:**  Explore the integration of MCP with Microsoft Entra ID for secure authorization.

---

### [