Azure AI Voicelive Dotnet

1---
2name: azure-ai-voicelive-dotnet
3description: |
4  Azure AI Voice Live SDK for .NET. Build real-time voice AI applications with bidirectional WebSocket communication. Use for voice assistants, conversational AI, real-time speech-to-speech, and voice-enabled chatbots. Triggers: "voice live", "real-time voice", "VoiceLiveClient", "VoiceLiveSession", "voice assistant .NET", "bidirectional audio", "speech-to-speech".
5package: Azure.AI.VoiceLive
6---
7 
8# Azure.AI.VoiceLive (.NET)
9 
10Real-time voice AI SDK for building bidirectional voice assistants with Azure AI.
11 
12## Installation
13 
14```bash
15dotnet add package Azure.AI.VoiceLive
16dotnet add package Azure.Identity
17dotnet add package NAudio                    # For audio capture/playback
18```
19 
20**Current Versions**: Stable v1.0.0, Preview v1.1.0-beta.1
21 
22## Environment Variables
23 
24```bash
25AZURE_VOICELIVE_ENDPOINT=https://<resource>.services.ai.azure.com/
26AZURE_VOICELIVE_MODEL=gpt-4o-realtime-preview
27AZURE_VOICELIVE_VOICE=en-US-AvaNeural
28# Optional: API key if not using Entra ID
29AZURE_VOICELIVE_API_KEY=<your-api-key>
30```
31 
32## Authentication
33 
34### Microsoft Entra ID (Recommended)
35 
36```csharp
37using Azure.Identity;
38using Azure.AI.VoiceLive;
39 
40Uri endpoint = new Uri("https://your-resource.cognitiveservices.azure.com");
41DefaultAzureCredential credential = new DefaultAzureCredential();
42VoiceLiveClient client = new VoiceLiveClient(endpoint, credential);
43```
44 
45**Required Role**: `Cognitive Services User` (assign in Azure Portal → Access control)
46 
47### API Key
48 
49```csharp
50Uri endpoint = new Uri("https://your-resource.cognitiveservices.azure.com");
51AzureKeyCredential credential = new AzureKeyCredential("your-api-key");
52VoiceLiveClient client = new VoiceLiveClient(endpoint, credential);
53```
54 
55## Client Hierarchy
56 
57```
58VoiceLiveClient
59└── VoiceLiveSession (WebSocket connection)
60    ├── ConfigureSessionAsync()
61    ├── GetUpdatesAsync() → SessionUpdate events
62    ├── AddItemAsync() → UserMessageItem, FunctionCallOutputItem
63    ├── SendAudioAsync()
64    └── StartResponseAsync()
65```
66 
67## Core Workflow
68 
69### 1. Start Session and Configure
70 
71```csharp
72using Azure.Identity;
73using Azure.AI.VoiceLive;
74 
75var endpoint = new Uri(Environment.GetEnvironmentVariable("AZURE_VOICELIVE_ENDPOINT"));
76var client = new VoiceLiveClient(endpoint, new DefaultAzureCredential());
77 
78var model = "gpt-4o-mini-realtime-preview";
79 
80// Start session
81using VoiceLiveSession session = await client.StartSessionAsync(model);
82 
83// Configure session
84VoiceLiveSessionOptions sessionOptions = new()
85{
86    Model = model,
87    Instructions = "You are a helpful AI assistant. Respond naturally.",
88    Voice = new AzureStandardVoice("en-US-AvaNeural"),
89    TurnDetection = new AzureSemanticVadTurnDetection()
90    {
91        Threshold = 0.5f,
92        PrefixPadding = TimeSpan.FromMilliseconds(300),
93        SilenceDuration = TimeSpan.FromMilliseconds(500)
94    },
95    InputAudioFormat = InputAudioFormat.Pcm16,
96    OutputAudioFormat = OutputAudioFormat.Pcm16
97};
98 
99// Set modalities (both text and audio for voice assistants)
100sessionOptions.Modalities.Clear();
101sessionOptions.Modalities.Add(InteractionModality.Text);
102sessionOptions.Modalities.Add(InteractionModality.Audio);
103 
104await session.ConfigureSessionAsync(sessionOptions);
105```
106 
107### 2. Process Events
108 
109```csharp
110await foreach (SessionUpdate serverEvent in session.GetUpdatesAsync())
111{
112    switch (serverEvent)
113    {
114        case SessionUpdateResponseAudioDelta audioDelta:
115            byte[] audioData = audioDelta.Delta.ToArray();
116            // Play audio via NAudio or other audio library
117            break;
118            
119        case SessionUpdateResponseTextDelta textDelta:
120            Console.Write(textDelta.Delta);
121            break;
122            
123        case SessionUpdateResponseFunctionCallArgumentsDone functionCall:
124            // Handle function call (see Function Calling section)
125            break;
126            
127        case SessionUpdateError error:
128            Console.WriteLine($"Error: {error.Error.Message}");
129            break;
130            
131        case SessionUpdateResponseDone:
132            Console.WriteLine("\n--- Response complete ---");
133            break;
134    }
135}
136```
137 
138### 3. Send User Message
139 
140```csharp
141await session.AddItemAsync(new UserMessageItem("Hello, can you help me?"));
142await session.StartResponseAsync();
143```
144 
145### 4. Function Calling
146 
147```csharp
148// Define function
149var weatherFunction = new VoiceLiveFunctionDefinition("get_current_weather")
150{
151    Description = "Get the current weather for a given location",
152    Parameters = BinaryData.FromString("""
153        {
154            "type": "object",
155            "properties": {
156                "location": {
157                    "type": "string",
158                    "description": "The city and state or country"
159                }
160            },
161            "required": ["location"]
162        }
163        """)
164};
165 
166// Add to session options
167sessionOptions.Tools.Add(weatherFunction);
168 
169// Handle function call in event loop
170if (serverEvent is SessionUpdateResponseFunctionCallArgumentsDone functionCall)
171{
172    if (functionCall.Name == "get_current_weather")
173    {
174        var parameters = JsonSerializer.Deserialize<Dictionary<string, string>>(functionCall.Arguments);
175        string location = parameters?["location"] ?? "";
176        
177        // Call external service
178        string weatherInfo = $"The weather in {location} is sunny, 75°F.";
179        
180        // Send response
181        await session.AddItemAsync(new FunctionCallOutputItem(functionCall.CallId, weatherInfo));
182        await session.StartResponseAsync();
183    }
184}
185```
186 
187## Voice Options
188 
189| Voice Type | Class | Example |
190|------------|-------|---------|
191| Azure Standard | `AzureStandardVoice` | `"en-US-AvaNeural"` |
192| Azure HD | `AzureStandardVoice` | `"en-US-Ava:DragonHDLatestNeural"` |
193| Azure Custom | `AzureCustomVoice` | Custom voice with endpoint ID |
194 
195## Supported Models
196 
197| Model | Description |
198|-------|-------------|
199| `gpt-4o-realtime-preview` | GPT-4o with real-time audio |
200| `gpt-4o-mini-realtime-preview` | Lightweight, fast interactions |
201| `phi4-mm-realtime` | Cost-effective multimodal |
202 
203## Key Types Reference
204 
205| Type | Purpose |
206|------|---------|
207| `VoiceLiveClient` | Main client for creating sessions |
208| `VoiceLiveSession` | Active WebSocket session |
209| `VoiceLiveSessionOptions` | Session configuration |
210| `AzureStandardVoice` | Standard Azure voice provider |
211| `AzureSemanticVadTurnDetection` | Voice activity detection |
212| `VoiceLiveFunctionDefinition` | Function tool definition |
213| `UserMessageItem` | User text message |
214| `FunctionCallOutputItem` | Function call response |
215| `SessionUpdateResponseAudioDelta` | Audio chunk event |
216| `SessionUpdateResponseTextDelta` | Text chunk event |
217 
218## Best Practices
219 
2201. **Always set both modalities** — Include `Text` and `Audio` for voice assistants
2212. **Use `AzureSemanticVadTurnDetection`** — Provides natural conversation flow
2223. **Configure appropriate silence duration** — 500ms typical to avoid premature cutoffs
2234. **Use `using` statement** — Ensures proper session disposal
2245. **Handle all event types** — Check for errors, audio, text, and function calls
2256. **Use DefaultAzureCredential** — Never hardcode API keys
226 
227## Error Handling
228 
229```csharp
230if (serverEvent is SessionUpdateError error)
231{
232    if (error.Error.Message.Contains("Cancellation failed: no active response"))
233    {
234        // Benign error, can ignore
235    }
236    else
237    {
238        Console.WriteLine($"Error: {error.Error.Message}");
239    }
240}
241```
242 
243## Audio Configuration
244 
245- **Input Format**: `InputAudioFormat.Pcm16` (16-bit PCM)
246- **Output Format**: `OutputAudioFormat.Pcm16`
247- **Sample Rate**: 24kHz recommended
248- **Channels**: Mono
249 
250## Related SDKs
251 
252| SDK | Purpose | Install |
253|-----|---------|---------|
254| `Azure.AI.VoiceLive` | Real-time voice (this SDK) | `dotnet add package Azure.AI.VoiceLive` |
255| `Microsoft.CognitiveServices.Speech` | Speech-to-text, text-to-speech | `dotnet add package Microsoft.CognitiveServices.Speech` |
256| `NAudio` | Audio capture/playback | `dotnet add package NAudio` |
257 
258## Reference Links
259 
260| Resource | URL |
261|----------|-----|
262| NuGet Package | https://www.nuget.org/packages/Azure.AI.VoiceLive |
263| API Reference | https://learn.microsoft.com/dotnet/api/azure.ai.voicelive |
264| GitHub Source | https://github.com/Azure/azure-sdk-for-net/tree/main/sdk/ai/Azure.AI.VoiceLive |
265| Quickstart | https://learn.microsoft.com/azure/ai-services/speech-service/voice-live-quickstart |
266