Azure AI Contentunderstanding Py

1---
2name: azure-ai-contentunderstanding-py
3description: |
4  Azure AI Content Understanding SDK for Python. Use for multimodal content extraction from documents, images, audio, and video.
5  Triggers: "azure-ai-contentunderstanding", "ContentUnderstandingClient", "multimodal analysis", "document extraction", "video analysis", "audio transcription".
6package: azure-ai-contentunderstanding
7---
8 
9# Azure AI Content Understanding SDK for Python
10 
11Multimodal AI service that extracts semantic content from documents, video, audio, and image files for RAG and automated workflows.
12 
13## Installation
14 
15```bash
16pip install azure-ai-contentunderstanding
17```
18 
19## Environment Variables
20 
21```bash
22CONTENTUNDERSTANDING_ENDPOINT=https://<resource>.cognitiveservices.azure.com/
23```
24 
25## Authentication
26 
27```python
28import os
29from azure.ai.contentunderstanding import ContentUnderstandingClient
30from azure.identity import DefaultAzureCredential
31 
32endpoint = os.environ["CONTENTUNDERSTANDING_ENDPOINT"]
33credential = DefaultAzureCredential()
34client = ContentUnderstandingClient(endpoint=endpoint, credential=credential)
35```
36 
37## Core Workflow
38 
39Content Understanding operations are asynchronous long-running operations:
40 
411. **Begin Analysis** — Start the analysis operation with `begin_analyze()` (returns a poller)
422. **Poll for Results** — Poll until analysis completes (SDK handles this with `.result()`)
433. **Process Results** — Extract structured results from `AnalyzeResult.contents`
44 
45## Prebuilt Analyzers
46 
47| Analyzer | Content Type | Purpose |
48|----------|--------------|---------|
49| `prebuilt-documentSearch` | Documents | Extract markdown for RAG applications |
50| `prebuilt-imageSearch` | Images | Extract content from images |
51| `prebuilt-audioSearch` | Audio | Transcribe audio with timing |
52| `prebuilt-videoSearch` | Video | Extract frames, transcripts, summaries |
53| `prebuilt-invoice` | Documents | Extract invoice fields |
54 
55## Analyze Document
56 
57```python
58import os
59from azure.ai.contentunderstanding import ContentUnderstandingClient
60from azure.ai.contentunderstanding.models import AnalyzeInput
61from azure.identity import DefaultAzureCredential
62 
63endpoint = os.environ["CONTENTUNDERSTANDING_ENDPOINT"]
64client = ContentUnderstandingClient(
65    endpoint=endpoint,
66    credential=DefaultAzureCredential()
67)
68 
69# Analyze document from URL
70poller = client.begin_analyze(
71    analyzer_id="prebuilt-documentSearch",
72    inputs=[AnalyzeInput(url="https://example.com/document.pdf")]
73)
74 
75result = poller.result()
76 
77# Access markdown content (contents is a list)
78content = result.contents[0]
79print(content.markdown)
80```
81 
82## Access Document Content Details
83 
84```python
85from azure.ai.contentunderstanding.models import MediaContentKind, DocumentContent
86 
87content = result.contents[0]
88if content.kind == MediaContentKind.DOCUMENT:
89    document_content: DocumentContent = content  # type: ignore
90    print(document_content.start_page_number)
91```
92 
93## Analyze Image
94 
95```python
96from azure.ai.contentunderstanding.models import AnalyzeInput
97 
98poller = client.begin_analyze(
99    analyzer_id="prebuilt-imageSearch",
100    inputs=[AnalyzeInput(url="https://example.com/image.jpg")]
101)
102result = poller.result()
103content = result.contents[0]
104print(content.markdown)
105```
106 
107## Analyze Video
108 
109```python
110from azure.ai.contentunderstanding.models import AnalyzeInput
111 
112poller = client.begin_analyze(
113    analyzer_id="prebuilt-videoSearch",
114    inputs=[AnalyzeInput(url="https://example.com/video.mp4")]
115)
116 
117result = poller.result()
118 
119# Access video content (AudioVisualContent)
120content = result.contents[0]
121 
122# Get transcript phrases with timing
123for phrase in content.transcript_phrases:
124    print(f"[{phrase.start_time} - {phrase.end_time}]: {phrase.text}")
125 
126# Get key frames (for video)
127for frame in content.key_frames:
128    print(f"Frame at {frame.time}: {frame.description}")
129```
130 
131## Analyze Audio
132 
133```python
134from azure.ai.contentunderstanding.models import AnalyzeInput
135 
136poller = client.begin_analyze(
137    analyzer_id="prebuilt-audioSearch",
138    inputs=[AnalyzeInput(url="https://example.com/audio.mp3")]
139)
140 
141result = poller.result()
142 
143# Access audio transcript
144content = result.contents[0]
145for phrase in content.transcript_phrases:
146    print(f"[{phrase.start_time}] {phrase.text}")
147```
148 
149## Custom Analyzers
150 
151Create custom analyzers with field schemas for specialized extraction:
152 
153```python
154# Create custom analyzer
155analyzer = client.create_analyzer(
156    analyzer_id="my-invoice-analyzer",
157    analyzer={
158        "description": "Custom invoice analyzer",
159        "base_analyzer_id": "prebuilt-documentSearch",
160        "field_schema": {
161            "fields": {
162                "vendor_name": {"type": "string"},
163                "invoice_total": {"type": "number"},
164                "line_items": {
165                    "type": "array",
166                    "items": {
167                        "type": "object",
168                        "properties": {
169                            "description": {"type": "string"},
170                            "amount": {"type": "number"}
171                        }
172                    }
173                }
174            }
175        }
176    }
177)
178 
179# Use custom analyzer
180from azure.ai.contentunderstanding.models import AnalyzeInput
181 
182poller = client.begin_analyze(
183    analyzer_id="my-invoice-analyzer",
184    inputs=[AnalyzeInput(url="https://example.com/invoice.pdf")]
185)
186 
187result = poller.result()
188 
189# Access extracted fields
190print(result.fields["vendor_name"])
191print(result.fields["invoice_total"])
192```
193 
194## Analyzer Management
195 
196```python
197# List all analyzers
198analyzers = client.list_analyzers()
199for analyzer in analyzers:
200    print(f"{analyzer.analyzer_id}: {analyzer.description}")
201 
202# Get specific analyzer
203analyzer = client.get_analyzer("prebuilt-documentSearch")
204 
205# Delete custom analyzer
206client.delete_analyzer("my-custom-analyzer")
207```
208 
209## Async Client
210 
211```python
212import asyncio
213import os
214from azure.ai.contentunderstanding.aio import ContentUnderstandingClient
215from azure.ai.contentunderstanding.models import AnalyzeInput
216from azure.identity.aio import DefaultAzureCredential
217 
218async def analyze_document():
219    endpoint = os.environ["CONTENTUNDERSTANDING_ENDPOINT"]
220    credential = DefaultAzureCredential()
221    
222    async with ContentUnderstandingClient(
223        endpoint=endpoint,
224        credential=credential
225    ) as client:
226        poller = await client.begin_analyze(
227            analyzer_id="prebuilt-documentSearch",
228            inputs=[AnalyzeInput(url="https://example.com/doc.pdf")]
229        )
230        result = await poller.result()
231        content = result.contents[0]
232        return content.markdown
233 
234asyncio.run(analyze_document())
235```
236 
237## Content Types
238 
239| Class | For | Provides |
240|-------|-----|----------|
241| `DocumentContent` | PDF, images, Office docs | Pages, tables, figures, paragraphs |
242| `AudioVisualContent` | Audio, video files | Transcript phrases, timing, key frames |
243 
244Both derive from `MediaContent` which provides basic info and markdown representation.
245 
246## Model Imports
247 
248```python
249from azure.ai.contentunderstanding.models import (
250    AnalyzeInput,
251    AnalyzeResult,
252    MediaContentKind,
253    DocumentContent,
254    AudioVisualContent,
255)
256```
257 
258## Client Types
259 
260| Client | Purpose |
261|--------|---------|
262| `ContentUnderstandingClient` | Sync client for all operations |
263| `ContentUnderstandingClient` (aio) | Async client for all operations |
264 
265## Best Practices
266 
2671. **Use `begin_analyze` with `AnalyzeInput`** — this is the correct method signature
2682. **Access results via `result.contents[0]`** — results are returned as a list
2693. **Use prebuilt analyzers** for common scenarios (document/image/audio/video search)
2704. **Create custom analyzers** only for domain-specific field extraction
2715. **Use async client** for high-throughput scenarios with `azure.identity.aio` credentials
2726. **Handle long-running operations** — video/audio analysis can take minutes
2737. **Use URL sources** when possible to avoid upload overhead
274